Jessica Renger
Biographic Data
| ID | 4411580 |
|---|---|
| NAME | Jessica Renger |
| GIVEN NAMES | Jessica |
| FAMILY NAME | Renger |
| SIGNATURE | RENGER J |
| AFFILIATIONS | Claremont Graduate University |
| ORCID | 0000-0002-5120-9504 |
| VERIFIED | Yes |
| TOTAL WORKS | 15 |
| TOTAL CITATIONS | 14 |
| AUTHOR COUNT | 15 |
| EDITOR COUNT | 0 |
| FIRST PUBLICATION YEAR | 2007 |
| LATEST PUBLICATION YEAR | 2026 |
| H-INDEX | 2 |
Participatory Meta-Evaluation
Despite being cemented in the Joint Committee Standards as a tenet of high-quality evaluation practice, use of meta-evaluation remains limited. One contributing factor to the limited utility of meta-evaluation may be that meta-evaluation is perceived as a burdensome process that necessitates a large degree of time, resources, and expertise to be completed successfully. To remedy this gap and embed meta-evaluation as a regular aspect of evaluation…
The Development and Implementation of an Interpersonal Skills Training Course for Young and Emerging Evaluators
Out of all the American Evaluation Association competency domains, evaluators report the least confidence in their interpersonal abilities and believe this is an area they need further training to develop. This study sought to remedy this gap by creating an interpersonal skills training course for young and emerging evaluators. A three-stage mixed methods design was employed to develop, pilot test, and implement a comprehensive interpersonal skil…
Applying the Renger System Test to Determine Whether a System Evaluation Is Fit for Evaluating a Complex Intervention
This article presents the Renger System Test (RST) as a method for assessing whether a system evaluation approach is suitable for evaluating complex interventions. The RST has three criteria: (1) the intervention includes multiple components, (2) these components operate interdependently, and (3) their interdependence produces an outcome that no single component can achieve alone. An NIH-funded case illustrates how to apply the RST to program sou…
Four suppositions about emergent properties of complex interventions that operate and function as systems
This article focuses on evaluating the emergent properties of complex interventions that operate and function as systems. After defining key terms, four suppositions of emergence are presented. The suppositions are based on the authors’ reflections on evaluating several complex interventions using a systems approach. Supposition 1 posits that there is an operational emergent property, paralleling the theory-driven evaluation concept of implementa…
A comparative case study illustrating the influence of a system perspective on the outcome evaluation of simple versus complex interventions
When evaluating the effectiveness of an intervention, the evaluation approach must match the intervention complexity, ensuring that the chosen evaluation is “fit for purpose.” For simple interventions, evaluating short-, mid-, and long-term outcomes is appropriate. However, for complex interventions, an additional outcome that must be considered is the essential system property that emerges as a result of the interaction of interdependent interve…
Evaluation Utility Metrics (EUMs) in Reflective Practice
The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. After an overview of reflective practice in evaluation, the different ways in which evaluators can hold themselves accountable are discussed. It is argued that reflective practice requires evaluators to go beyond evaluation quality (i.e., technical quality and methodological rigor) when assessing evaluation practice to include…
New Insights for Tracking and Reporting Milestones
Despite being a common evaluation metric, milestones are poorly defined, and guidance for their use is underdeveloped in the evaluation literature. This practice note attempts to address this oversight by providing evaluation-specific guidance for tracking and reporting milestones. Using a case study, we share our initial approach to tracking and reporting milestones, as well as how critical reflection guided by the evaluation standards provided …
The Prevalence of Evaluator Anxiety in Practice
Anxiety is a multifaceted force that can negatively impact the ability of evaluators to succeed in practice. In the evaluation literature, discussions concerning anxiety have primarily been limited to strategies to reduce stakeholder anxiety to encourage positive and productive working relationships with evaluators. This study was among the first to explicitly recognize and empirically assess the anxiety evaluators feel during practice. Using a m…
Using the Homeland Security Exercise and Evaluation Program (HSEEP) Building Block Approach to Implement System Evaluation Theory (SET)
This article shares lessons learned in applying system evaluation theory (SET) to evaluate a Clinical and Translational Research Center (CTR) funded by the National Institutes of Health. After describing how CTR support cores are intended to work interdependently as a system, the case is made for SET as the best fit for evaluating this evaluand. The article then details how the evaluation was also challenged to facilitate a CTR culture shift, hel…
Comparing and Contrasting a Program versus System Approach to Evaluation
This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective. The terms program, systems, systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes.…
Lessons learned in evaluating the infrastructure of a Centre for Translational Research
This paper shares lessons learned while evaluating the implementation of a Clinical and Translational Research Center (CTR). To meet its overarching goals the CTR consists of numerous research support units (e.g., biostats, community engagement, professional development, etc.) that are intended to work together collaboratively. It is then argued that an evaluation approach grounded in system thinking was the best fit to evaluate this key CTR desi…
The connection between logic models and systems thinking concepts
This article examines the relationship between systems thinking concepts and the logic model. Two notable shortcomings of the logic model are illustrated: depicting the program theory linearly and failing to place the program in context. Both issues lead to an artificial depiction of reality making evaluation findings difficult to interpret. Systems thinking concepts are defined and how they address these shortcomings is discussed. This article t…
Defining Systems to Evaluate System Efficiency and Effectiveness
This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no l…
Evaluating System Cascading Failures
This article shares methods used to evaluate system cascading failures. A cascading failure occurs when a problem is passed from one subsystem to a downstream subsystem creating a domino effect that undermines system efficiency and effectiveness. First, the basics of system evaluation theory (SET) are reviewed. Then drawing on different examples from the evaluation of emergency response systems the article describes how a) standard operating proc…
What an Eight-Year-Old Can Teach Us About Logic Modelling and Mainstreaming
This article presents a short case narrative, the purpose of which is to illustrate that complex evaluation methodologies such as logic modelling can be simplified to the point where a child can be guided through the process quickly. However, the case narrative also serves to highlight the potential consequences to program development and evaluation activities when the process is oversimplified. Like a double-edged sword, simplifying the process …
Defining Systems to Evaluate System Efficiency and Effectiveness
This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no l…
Evaluating System Cascading Failures
This article shares methods used to evaluate system cascading failures. A cascading failure occurs when a problem is passed from one subsystem to a downstream subsystem creating a domino effect that undermines system efficiency and effectiveness. First, the basics of system evaluation theory (SET) are reviewed. Then drawing on different examples from the evaluation of emergency response systems the article describes how a) standard operating proc…
A comparative case study illustrating the influence of a system perspective on the outcome evaluation of simple versus complex interventions
When evaluating the effectiveness of an intervention, the evaluation approach must match the intervention complexity, ensuring that the chosen evaluation is “fit for purpose.” For simple interventions, evaluating short-, mid-, and long-term outcomes is appropriate. However, for complex interventions, an additional outcome that must be considered is the essential system property that emerges as a result of the interaction of interdependent interve…
Lessons learned in evaluating the infrastructure of a Centre for Translational Research
This paper shares lessons learned while evaluating the implementation of a Clinical and Translational Research Center (CTR). To meet its overarching goals the CTR consists of numerous research support units (e.g., biostats, community engagement, professional development, etc.) that are intended to work together collaboratively. It is then argued that an evaluation approach grounded in system thinking was the best fit to evaluate this key CTR desi…
The Prevalence of Evaluator Anxiety in Practice
Anxiety is a multifaceted force that can negatively impact the ability of evaluators to succeed in practice. In the evaluation literature, discussions concerning anxiety have primarily been limited to strategies to reduce stakeholder anxiety to encourage positive and productive working relationships with evaluators. This study was among the first to explicitly recognize and empirically assess the anxiety evaluators feel during practice. Using a m…
The connection between logic models and systems thinking concepts
This article examines the relationship between systems thinking concepts and the logic model. Two notable shortcomings of the logic model are illustrated: depicting the program theory linearly and failing to place the program in context. Both issues lead to an artificial depiction of reality making evaluation findings difficult to interpret. Systems thinking concepts are defined and how they address these shortcomings is discussed. This article t…
What an Eight-Year-Old Can Teach Us About Logic Modelling and Mainstreaming
This article presents a short case narrative, the purpose of which is to illustrate that complex evaluation methodologies such as logic modelling can be simplified to the point where a child can be guided through the process quickly. However, the case narrative also serves to highlight the potential consequences to program development and evaluation activities when the process is oversimplified. Like a double-edged sword, simplifying the process …
Defining Systems to Evaluate System Efficiency and Effectiveness
This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no l…
Evaluating System Cascading Failures
This article shares methods used to evaluate system cascading failures. A cascading failure occurs when a problem is passed from one subsystem to a downstream subsystem creating a domino effect that undermines system efficiency and effectiveness. First, the basics of system evaluation theory (SET) are reviewed. Then drawing on different examples from the evaluation of emergency response systems the article describes how a) standard operating proc…
The connection between logic models and systems thinking concepts
This article examines the relationship between systems thinking concepts and the logic model. Two notable shortcomings of the logic model are illustrated: depicting the program theory linearly and failing to place the program in context. Both issues lead to an artificial depiction of reality making evaluation findings difficult to interpret. Systems thinking concepts are defined and how they address these shortcomings is discussed. This article t…
Comparing and Contrasting a Program versus System Approach to Evaluation
This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective. The terms program, systems, systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes.…
Lessons learned in evaluating the infrastructure of a Centre for Translational Research
This paper shares lessons learned while evaluating the implementation of a Clinical and Translational Research Center (CTR). To meet its overarching goals the CTR consists of numerous research support units (e.g., biostats, community engagement, professional development, etc.) that are intended to work together collaboratively. It is then argued that an evaluation approach grounded in system thinking was the best fit to evaluate this key CTR desi…
Using the Homeland Security Exercise and Evaluation Program (HSEEP) Building Block Approach to Implement System Evaluation Theory (SET)
This article shares lessons learned in applying system evaluation theory (SET) to evaluate a Clinical and Translational Research Center (CTR) funded by the National Institutes of Health. After describing how CTR support cores are intended to work interdependently as a system, the case is made for SET as the best fit for evaluating this evaluand. The article then details how the evaluation was also challenged to facilitate a CTR culture shift, hel…
Evaluation Utility Metrics (EUMs) in Reflective Practice
The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. After an overview of reflective practice in evaluation, the different ways in which evaluators can hold themselves accountable are discussed. It is argued that reflective practice requires evaluators to go beyond evaluation quality (i.e., technical quality and methodological rigor) when assessing evaluation practice to include…
New Insights for Tracking and Reporting Milestones
Despite being a common evaluation metric, milestones are poorly defined, and guidance for their use is underdeveloped in the evaluation literature. This practice note attempts to address this oversight by providing evaluation-specific guidance for tracking and reporting milestones. Using a case study, we share our initial approach to tracking and reporting milestones, as well as how critical reflection guided by the evaluation standards provided …
The Prevalence of Evaluator Anxiety in Practice
Anxiety is a multifaceted force that can negatively impact the ability of evaluators to succeed in practice. In the evaluation literature, discussions concerning anxiety have primarily been limited to strategies to reduce stakeholder anxiety to encourage positive and productive working relationships with evaluators. This study was among the first to explicitly recognize and empirically assess the anxiety evaluators feel during practice. Using a m…
Four suppositions about emergent properties of complex interventions that operate and function as systems
This article focuses on evaluating the emergent properties of complex interventions that operate and function as systems. After defining key terms, four suppositions of emergence are presented. The suppositions are based on the authors’ reflections on evaluating several complex interventions using a systems approach. Supposition 1 posits that there is an operational emergent property, paralleling the theory-driven evaluation concept of implementa…
A comparative case study illustrating the influence of a system perspective on the outcome evaluation of simple versus complex interventions
When evaluating the effectiveness of an intervention, the evaluation approach must match the intervention complexity, ensuring that the chosen evaluation is “fit for purpose.” For simple interventions, evaluating short-, mid-, and long-term outcomes is appropriate. However, for complex interventions, an additional outcome that must be considered is the essential system property that emerges as a result of the interaction of interdependent interve…
Applying the Renger System Test to Determine Whether a System Evaluation Is Fit for Evaluating a Complex Intervention
This article presents the Renger System Test (RST) as a method for assessing whether a system evaluation approach is suitable for evaluating complex interventions. The RST has three criteria: (1) the intervention includes multiple components, (2) these components operate interdependently, and (3) their interdependence produces an outcome that no single component can achieve alone. An NIH-funded case illustrates how to apply the RST to program sou…
Participatory Meta-Evaluation
Despite being cemented in the Joint Committee Standards as a tenet of high-quality evaluation practice, use of meta-evaluation remains limited. One contributing factor to the limited utility of meta-evaluation may be that meta-evaluation is perceived as a burdensome process that necessitates a large degree of time, resources, and expertise to be completed successfully. To remedy this gap and embed meta-evaluation as a regular aspect of evaluation…
The Development and Implementation of an Interpersonal Skills Training Course for Young and Emerging Evaluators
Out of all the American Evaluation Association competency domains, evaluators report the least confidence in their interpersonal abilities and believe this is an area they need further training to develop. This study sought to remedy this gap by creating an interpersonal skills training course for young and emerging evaluators. A three-stage mixed methods design was employed to develop, pilot test, and implement a comprehensive interpersonal skil…
Computer Science (13 works) · Evaluation and Performance Assessment (13 works) · Engineering (12 works) · Health Policy Implementation Science (9 works) · Management science (9 works) · Psychology (7 works) · Artificial Intelligence (5 works) · Medicine (5 works) · Process management (5 works) · Risk analysis (engineering (5 works)