Evaluate Peer Support

Please see Disclaimer: Peer Support Resources below.

Organizations, program managers, and other people who manage peer support programs are often responsible for monitoring and evaluating them. Evaluation may include monitoring the experience of peer supporters, tracking the programs they deliver, measuring the satisfaction and experience of people with diabetes, and determining how behaviors, quality of life, and the overall health of people with diabetes improve upon participating in the programs.

Evaluation can take on many forms and levels, from tracking what you are doing to determining how it is making a difference. You may want to:

• Evaluate the peer support training
• Monitor the interaction between peer supporters and participants
• Evaluate how the program is implemented (Process Evaluation)
• Evaluate behavior changes (Impact Evaluation)
• Evaluate changes in health status or clinical outcomes (Outcome Evaluation)

In this section, you will find information and resources on how to evaluate a peer support program.

• Learn about planning an evaluation
• Learn about Process Evaluation
• Learn about Impact and Outcome Evaluation
• Find examples of evaluations conducted on peer support programs
• Find program evaluation tools

Planning Your Evaluation: RE-AIM Model

This evaluation framework (outlined below) includes and goes beyond a sole focus on the “did it work?” question to help expand our understanding of and translate effective health promotion and behavior change interventions.

According to RE-AIM, evaluations should examine five dimensions:

• Reach into the target population;
• Efficacy or effectiveness of the intervention;
• Adoption by target settings or institutions;
• Implementation-consistency of delivery of intervention; and
• Maintenance of intervention effects in individuals and populations over time.

Researchers and community leaders can visit the RE-AIM website for additional background and resources for evaluation.

Learn more about the RE-AIM Model by viewing this presentation: http://conferences.thehillgroup.com/conferences/di2007/dayone/02_Glasgow.pdf

View the RE-AIM Planning Tool to help in planning your evaluation.

In addition to RE-AIM, the following resources may also be helpful for a comprehensive approach to evaluation:

• The World Health Organization site provides a Workbook on Planning Evaluation, Implementing Evaluation, and Needs Assessment Evaluation.
• The University of Kansas Community Tool Box offers a section on Developing an Evaluation Plan.
• The W. K. Kellogg Foundation outlines a blueprint for conducting project evaluations.

[back to top]

Process Evaluation

Process evaluation assesses how a program is implemented. It takes into account all program inputs (e.g. theoretical foundations, goals and objectives, resources, etc.), activities (e.g. training, peer support interactions, etc.), and reactions of participants and stakeholders. Some areas you might consider assessing during a process evaluation are:

• Context: what are aspects of the community or environment that might influence the program?
• Reach: who is participating?
• Dose Delivered: how are peer supporters trained?
• Dose Received: what happens in interactions between peers supporters and participants?
• Fidelity: was the program delivered as planned (training and peer support interaction)?

Some ways you might measure process evaluation questions are:

• Interviews or questionnaires with peer supporters, participants, or stakeholders
• Observation of trainings or peer support interactions
• Administrative bookkeeping

• The World Health Organization provides a Workbook on Process Evaluation.
• Process Evaluation for Public Health Interventions and Research is a book chapter by Linnan and Steckler providing an overview of process evaluation.
• Section III.e of this Peer Support Training Curriculum provides insight regarding evaluating peer training.
• This article of peer support and smoking among youth outlines a comprehensive process evaluation approach for detecting change.
• Appendix 17 of Peer Support Resource Manual provides sample process evaluation tools and other resources.
• The Georgia Department of Human Resources, Division of Public Health, provides a workbook for designing process evaluations.
• Interested in using a focus group as part of a process evaluation? Read this step-by-step guide to conducting focus groups.
• Chapter 3 of the Community Toolbox, Assessing Community Needs and Resources, describes focus groups and other needs assessment strategies.
• This University of Illinois report describes a process evaluation of a peer certification training program.

[back to top]

Impact and Outcome Evaluation

According to a 2007 WHO report (http://www.who.int/diabetes/publications/Diabetes_final_13_6.pdf), there are several key indicators of the effectiveness of a peer support program. These indicators of success can be broken down into impact and outcome evaluation measures.

Impact Evaluation: An impact evaluation assesses the immediate effect a program has on target behaviors and their influencing factors (internal or external). Indicators identified by the WHO report that fall in this category are:

• Self-reported quality of life and emotional distress
• Adherence to behavioral and medication prescriptions
• Knowledge, attitudes, self-efficacy, autonomy and ability to function in their life roles and at work or school

Tools that can be used to assess behavioral measures include:

• Revised Summary of Diabetes Self-Care Activities Scale (SDSCA)
• Perceived Diabetes Self-Management Scale (PDSMS)
• Self-Efficacy for Diabetes Management
• Diabetes Support Scale (DSS) (in appendix of article)
• Diabetes Empowerment Scale (outlined in Table 3)

Tools that are often used to assess Quality of Life include:

• EQ-5D (EuroQoL)
• WHO Quality of Life Scale - brief; can be found on p. 73-76 of the WHO Quality of Life Study
• Diabetes Distress Scale (in English and Spanish)
• Patient Health Questionnaire, also known as the PHQ-9 (Depression)

Outcome Evaluation: An outcome evaluation assesses the effect of a program on health status and quality-of-life indicators.

Among the clinical parameters identified by the WHO report:

• Presence or absence of symptoms (hypo or hyperglycemia)
• HbA1C
• Cardiovascular risk factor control
• Hospitalizations and emergency room visits
• Health care resource consumption
• Presence or development of long term diabetes or cardiovascular complications
• Expenses (costs) and savings (benefits) associated with the program

Additional indicators that are being used to evaluate program outcomes include:

• BMI (Body Mass Index)
• Cholesterol
• Blood Pressure

• The World Health Organization provides a Workbook on Outcome Evaluation, Cost Evaluation, Client Satisfaction Evaluation, and Economic Evaluation.
• The Free Management Library provides a Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very Limited Resources. The document provides guidance toward basic planning and implementation of an outcomes evaluation in nonprofit organizations.
• This evaluation article reports on peer support in type 2 diabetes among adults in Ireland (2007).
• This evaluation article reports on an evaluation approach for examining Aboriginal Diabetes Prevention and Care (including peer support).

[back to top]

Examples of Peer Support Evaluations

Evaluation of a Pilot Training Program for Community Health Workers (CHW)

This report is a summary of several sources of data used to evaluate the Community Voices Miami Community Health Worker Project Level I Training Pilot. The report is structured to meet two main objectives of the program: (1) informing the future direction of the curriculum, and (2) reporting evaluation results to funding agencies. http://www.communityvoicesmiami.org/Reports/other/otherreport14.htm

Evaluation of Peer Support Services (Modernization Initiative on Kidney Disease)

Appendix ii of this report outlines the central role of the steering group in designing and carrying out the evaluation, with assistance from an independent evaluator. The approach was participatory, iterative and action-oriented to ensure that the evaluation added value by building on what was already known from other sources and answering questions that were important to those responsible for managing peer support services. The evaluation was not an ‘add on' piece of research, but a facilitated process in a number of stages, in which the steering group framed questions; planned and participated in data gathering and analysis; received feedback as the work progressed; and contributed to interpreting the findings. The evaluation was carried out in three phases. In the first phase a group discussion was held with peer support volunteers, facilitated by the independent evaluator. The findings from this phase helped the steering group plan the second phase, interviews with a sample of patients who had received peer support. http://www.gsttcharity.org.uk/pdfs/whitecoat.pdf

Peer Education Training Sites Evaluation

The Peer Education Evaluation and Resource Center (PEER Center) helps plan and implement sustainable peer programs for Peer Education Training Sites:

• (i) Center for Health Training in Oakland, CA
• (ii) Harlem Hospital in New York, NY;
• (iii) American Red Cross St. Louis Area Chapter;
• (iv) KC Free Clinic in Kansas City, MO; and
• (v) Midwest AIDS Training and Education Center.

In addition, the PEER Center coordinates training and capacity building work. This Center is a national resource and evaluation center for people living with HIV and organizations interested in PEER education training programs.

Peer Outcomes Protocol Project

This protocol administration manual was developed as a way to evaluate community-based, mental health peer support programs. Each module in the manual describes how to conduct interviews, use questionnaires, and analyze the data collected in order to better focus on improving quality of life and peer supports for people with psychiatric disabilities.

The National Diabetes Program Evaluation Framework

This guide describes how to design an evaluation of a multifaceted public health education program. This framework has helped program planners and evaluators develop measurable short-term and long-term outcomes.

Depression and Bipolar Support Alliance

This website provides a report of their Evaluation of the Peer Support Certification Training Program.

The Results of an Evaluation of the Peer Support Program

The Nova Institution for Women provides a report outlining their evaluation of the peer support program which incorporate both quantitative and qualitative analyses in the program assessment.

[back to top]

Program Evaluation Tools

As with building a house, a good evaluation is all about having the right tools. Many program managers choose to create their own evaluation tools (e.g. questionnaires, interview guides) from scratch, while others adapt already well-tested and well-used tools to better meet the needs of their setting or population. You may view examples of evaluation and assessment tools below.

• NEW- The Consensus Evaluation provides a core set of shared measures of benefits of self management and peer support in diabetes identified and used by our Program Development Center and Evaluation Grantees.
• The Diabetes at Work project provides a set of common tools used for program evaluation.
• This 2006 article from Prevention Chronic Disease describes methods and approaches to program evaluation.
• Section K: Program Evaluation (pg. 57) of the Mentoring Partnership Program Manual describes how to develop a plan for program evaluation.
• Chapter 6 of this Peer Transitions Project Report using youth leadership outlines important program evaluation components for a peer program.
• Annex 2 (pg. 144) and Annex 3 (pg. 147) of this peer mentor training manual include example pre- and post-training tests to rate the quality of the training and also peer educator and trainer evaluation forms.
• The IPAS Train-the-Trainer manual includes several evaluation tools, including course evaluation, trainer evaluation, daily evaluation, and participant evaluation forms. These can be found in Appendices 3, 4, 8 and 9.
• Appendix 1 (pg. 176) of this peer supporter training manual includes a checklist for observers evaluating peer supporters in training.
• The Robert Woods Johnson Foundation Diabetes Initiative provides resources on project participant assessment, pre-test and post-test questionnaires and other program evaluation tools
• The University of Kansas Community Tool Box provides a number of evaluation resources including Evaluating Community Programs and Initiatives, Developing Training Programs for Volunteers and Evaluating the Trainees, and a Trainee Evaluation Form and Checklist.
• The U.S Centers for Disease Control and Prevention Evaluation Working Group provides a host of descriptive information and practical tools for a program evaluation framework.

[back to top]