Program Development Guide
Selecting an Evaluation Type
In this Chapter:
1. Selecting an Evaluation Type
Selecting an Evaluation Type
Program evaluation is necessary to answer the questions “did peer support work,” and “how did it work.” Choosing an evaluation type largely depends on the question you are trying to answer. Below are four types of evaluation – process, outcome, impact, and quality – that are commonly utilized by health practitioners and interventionists. Program Managers may choose to conduct some or all of the four types of evaluation of their programs. Conducting all of the evaluation types is most beneficial to get a good picture of how the program worked, whether or not it achieved its goals, and how it impacted both staff and participants. It is important to note that allotting appropriate staff time and resources to conducting evaluation activities is a key component of a successful evaluation.
Process Evaluation – How was peer support delivered?
Process evaluation assesses how a program is implemented and takes into account all program inputs (e.g., theoretical foundations, goals and objectives, resources, etc.), activities (e.g., training, peer support interactions, etc.), and reactions of participants and stakeholders. Some areas you might consider assessing during a process evaluation are:
- Context: What aspects of the community or environment that influence the program?
- Reach: Who is participating in the program?
- Dose delivered: How are peer supporters trained?
- Dose received: What happens in interactions between peers supporters and participants?
- Fidelity: Was the program delivered as planned (training and peer support interaction)?
Here is a suggested reading for process evaluation: Linnan, L., and Steckler A. (2002). Process evaluation and public health interventions: An overview. In: Steckler, A., and Linnan, L (Eds.), Process Evaluation in Public Health Interventions and Research. San Francisco: Jossey-Bass Publishers. (pg. 1–23).
Outcome Evaluation – Did peer support have an effect on health status?
Outcome evaluations assess short-term or immediate changes that have occurred as a result of an intervention. Outcome evaluations compare differences between baseline and measures at completion of a one-time program or at specified intervals (e.g., every year, every 3 years, or every 5 years) of an ongoing program. Questions asked in an outcome evaluation might include:
- What effect is the intervention having on its stakeholders or participants (e.g., changes in knowledge, attitudes, or behavior)?
- What unexpected outcomes, if any, have resulted from the intervention?
- What can be modified to make the intervention more effective?
- Is there any evidence showing that funders should continue to support this intervention?
Impact Evaluation – Did peer support have an impact on targeted health behaviors?
Impact evaluations assess the long-term changes in target behaviors that can be directly attributed to a particular intervention. The central question in an impact evaluation is: What would have happened to those receiving the intervention if they had not been involved with the program? Therefore, impact evaluations are dependent on developing a comparison group that is similar as possible (in observable and unobservable dimensions) to those receiving the intervention. This comparison allows for the establishment of causality (i.e., attributing observed changes in targeted health behaviors to the intervention) while removing internal and external complicating factors. Some areas you might consider assessing during an impact evaluation are:
- Is the intervention working?
- What parts of the intervention are working?
- Should the intervention be redesigned?
- Should the intervention be scaled up?
Quality Improvement Evaluation – How can we improve peer support interventions?
Quality Improvement (QI) evaluations assess ongoing adherence to intervention guidelines, efficient use of intervention resources, and improving coordination between intervention components. Rather than proving something to be effective or not, the goal of QI is to continuously improve interventions in the settings they operate and for the populations they serve. The QI concept has been well-recognized as a way to address population needs, patient care issues, and health system challenges (e.g., high resource utilization). Related methodologies such as Plan-Do-Study-Act (PDSA) promoted by the Institute for Healthcare Improvement (IHI) produced dynamic, rapid feedback for interventionists and practitioners.
Promoting Quality Improvement in Your Organization
Leaders and managers of peer support programs can promote QI activities in the following ways:
- Create a vision for quality by setting shared goals for performance
- Build staff capacity for QI by providing essential training and tools for QI
- Motivate staff for QI by communicating to them that improvements are possible and welcomed, and encouraging them to make quality part of their job.
- Establish a QI team involving representatives from all departments to the QI process
- Dedicate time to measure performance of the program
- Make sure that “feedback” collected from the surveys, interviews is heard and incorporated into the program
- Provide time to openly discuss both successes and failures
- Use available existing resources to strengthen QI
- Include a budget for QI activities
World Health Organization (2008). Operations manual for delivery of HIV prevention, care and treatment at primary health centers in high-prevalence, resource-constrained settings. Chapter 11: Quality Improvement.
© 2015 | Peers for Progress