Program Development Guide
In this Chapter:
5. Data Collection
Whether it is for M&E quality improvement, the collection of information/data plays an essential role in helping convey a well-rounded picture for the evaluation of peer support programs. The quality of data directly impacts the credibility of the evaluation, as the information would be used to strengthen the evaluation judgments and the recommendations that follow from them. Therefore, it is important to collect information that is reliable and relevant for answering stakeholders’ questions about the evaluation and the peer support program. These are important points to convey to peer supporters as well, since they may be providing some of the data or collecting data.
Information/data collected in a systematic way can significantly impact quality improvements process, help save costs, and contribute to provide credible clarity to stakeholders. Although each type of data has its limitation, an evaluation’s overall credibility can be improved by using different type of data for an evaluation and employing various procedures for data collection, analysis and interpretation. In addition, involving stakeholders in defining and gathering data can also enhance perceived credibility because they will be more likely accept the evaluation’s conclusions and act on the its recommendations.
When collecting data for evaluation, it is important to consider the following:
- Type of information needed for the evaluation (e.g., quantitative information, qualitative information, or both)
- Design of the data collection plan (e.g., data collection tools/ methodology)
- Quality assurance for data collection (e.g., measure indicators, costs, accuracy)
1. Type of Data
Information or facts collected for evaluation are commonly divided into two types:
Quantitative data is information/facts about quantities that can be measured and written down in numbers. The data
- Answers the questions: “How many?”, “How often?”, or “What proportion?”
- Measures levels of behavior and trends.
- Is objective, standardized, and relatively easily analyzed.
- Is comparable to similar data from other communities and levels that use similar measures
Qualitative data includes information/facts that are about qualities or characteristics and can’t be measured easily in numeric form. The data
- Answers the questions: “Why?” “Why not?” or “What does it mean?”
- Allows insight into behavior, trends, and perceptions.
- Is subjective and explanatory.
- Helps interpret quantitative data, provides depth of understanding.
2. Data Collection Instruments/Tools
Data can be collected through a number of data collection instruments, including computer-aided personal or telephone interview, mailed questionnaires, electronic or internet questionnaires and direct observation.
Data can be collected using different methodologies, including:
- Sample survey
- Administrative data
- Tracer study
- Focus group
- Case study
3. Quality Criteria for Data Collection
When collecting data based on pre-determined measurable indicators, it is important to consider the following criteria to ensure quality of the collected data:
- Accuracy – Is the data collected correct and reliable?
- Efficiency – Is the tool used to collect the data the most economic way to achieve the objectives?
- Effectiveness – Have desired objectives been achieved?
- Feasibility and timeliness – Can the data be collected in a cost-effective way, and provide information in a timely manner?
- Relevance – What is the relevance of the information to stakeholders?
- Security – Does the data collection process follow confidentiality protocol?
- Utility – Does the collected facts provide the right information to answer the questions posed?
|Northwest Center for Public Health Practice’s Data Collection for Program Evaluation||This toolkit goes through different data collection methods (document review, observation, survey, interview, and focus group) and provides an overview, advantages, disadvantages, examples, and further resources for each method.|
|American Society for Quality’s Data Collection and Analysis Toolkit||This includes a number of tools to collect or analyze data|
|Pell Institute’s Evaluation Toolkit||The evaluation toolkit provides a summary of the most commonly used methods in evaluation. It also includes extra information as well as further resources and examples.|
|FHI 360’s Qualitative Research Methods: A Data Collector’s Field Guide||This is a comprehensive guide designed to be used as a tool for training data collectors in qualitative data collection methods and data management. It is divided into five modules: (1) Qualitative Research Methods Overview, (2) Participant Observation, (3) In-Depth Interviews, (4) Focus Groups, and (5) Data Documentation and Management|
|RAND’s Data Collection Methods: Semi-Structured Interviews and Focus Groups||This course covers an overview of two types of qualitative data collection methodologies: semi-structured interviews and focus groups. It provides an overview of these methods, as well as some practical tools that will allow readers to improve their qualitative data collection.|
|American Red Cross’ Monitoring and Evaluation Planning Guidelines and Tools (pg. 11–14 and Annex IV)||This guide provides concise guidance to readers to develop a comprehensive M&E system for international humanitarian relief and development programs.|
|Agency for Healthcare Research and Quality’s Improving Data Collection across the Health Care System||This page includes case studies on data collection at different health care settings.|
|Institute for Health Improvement (IHI)’s Simple Data Collection Planning||Simple Data Collection Planning is a process tool that ensure the data collected for performance improvement is useful and reliable, without unnecessary cost and time|
|IHI’s Sampling Tool||Adapted from the IHI’s Methods and Tools for Breakthrough Improvement course, sampling has been used by hundreds of health care organizations.|
|NY State Department of Health AIDS Institute’s NQC Quality Academy: Collecting Performance Data||National Quality Center’s material provides tutorial on how to effectively and efficiently collect quality data and translate it into QI activities. It includes examples of sampling records for performance reviews by establishing review eligibility criteria, identifying minimal sample sizes, and selecting a random sample.|
|HIVQUAL Project Sampling Methodology||This step-by-step guide, developed by the National HIVQUAL Project, helps health care providers determine how many patient records to review for accurate performance measurement and how to effectively sample records.|
|University of Iowa Hospital & Clinics’ Data Collection Plan Worksheet & Template||This tool services as a reminder of why data collection is importance and reinforces standard collection methods for sharing among team members.|
|Provider and Staff Satisfaction Survey||This survey, developed by IHI, provides a template to test providers and staff satisfaction within an organization.|
|Consumer Assessment of Healthcare Providers and System (CAHPS)||The CAHPS program is a public-private initiative to develop standardized survey of patients’ experiences with ambulatory and facility-level care. A CAHPS Survey and Reporting Kit explains how to prepare and field a CAHPS questionnaire, analyze the results, and produce consumer-friendly reports.|
|IHI’s Short Survey Tool||Short surveys provide simple and prompt feedback to assess whether attempts to improve performance are going in the right direction. They can also be used to pinpoint certain areas of interest (e.g., did patients find the new form easy to understand?).|
© 2015 | Peers for Progress