Impact Measurement in Monitoring and Evaluation

Explore impact evaluation methods to attribute changes to the intervention by constructing a counterfactual, addressing causality. Compare experimental and quasi-experimental designs, including RCT, differences-in-differences, PSM, RDD, and ethical considerations.

Master core terms like counterfactual, control and treatment groups, selection bias, and internal and external validity. Explore A and B testing and randomized controlled trials as foundations for impact evaluation.

Explore randomized controlled trials by randomly assigning participants to treatment or control, ensuring baseline equivalence of observable and unobservable characteristics, and estimating impact from outcome comparisons.

Discover how randomization in randomized controlled trials assigns eligible beneficiaries to treatment and control groups, ensuring similar observable and non-observable characteristics so only the treatment affects outcomes.

Explore how randomized controlled trials achieve randomization using rotary method, facing method, and randomized encouragement, and how interventions, controls, groups, and sms prompts unfold.

Describe the rotary method of randomization, selecting a fair, equal-chance subset of eligible beneficiaries for intervention and using the rest as a control group in resource-limited programs.

Explore randomization encouragement, where everyone can access an intervention but only some are randomly encouraged, such as discounts or SMS reminders. Compare treated and control groups to test effects.

Choose the unit of randomization as an individual or a cluster, applying treatment or control at the respective level, and adjust sample size for clustering.

Develop a clear research question and define the unit of assignment, whether individuals or clusters. Randomly assign treatment and control, implement the program, and measure impact by comparing outcomes.

Compare treatment and control groups under identical conditions to estimate the true impact as the difference between treatment and comparison outcomes, with randomization ensuring fairness and causal attribution.

Learn the seven RCT assumptions for valid impact evaluation, including proper random assignment, no spillovers, stable balance, sufficient power, compliance, and consistent counterfactual conditions.

Compare internal and external validity, showing how internal validity measures the true impact of the intervention, while external validity assesses generalization to the population via sample representativeness.

Identify why randomized controlled trials may not apply, due to ethical or legal constraints, stakeholder resistance, contamination risks, large-scale policies, pandemics, or interventions, and consider mixed method or quasi experiments.

An RCT example shows tutoring's impact on student test scores, delivering a 13-point post-test gain (85 vs 72) with randomization ensuring baseline comparability and confounding reduction.

Explore difference-in-differences, a quasi-experimental method to estimate causal effects from observational data by comparing outcome changes over time between treatment and control groups under a parallel trend assumption.

The parallel trend assumption states that, absent treatment, treatment and control differences stay constant over time, enabling difference-in-differences to estimate the causal effect using prior data.

Apply the basic difference-in-differences setup by comparing baseline and post-intervention changes in treatment and control groups to estimate the intervention's impact.

Explore a real-life difference-in-differences example measuring a training program's impact on average scores by comparing treatment and control groups with baseline and post-intervention data.

Identify the five core assumptions of difference-in-differences: parallel trends, no spillover, stable composition, no simultaneous intervention, and similar external shocks.

Apply the difference-in-differences method to evaluate a job training program’s impact on wages, using pre and post intervention data for treatment and control groups under parallel trends.

Explore propensity score matching (PSM) to create a counterfactual comparison group when randomization isn't feasible by matching treatment units with similar non-treatment units on observable characteristics, reducing bias.

Learn to match treatment and untreated units in impact measurement using observable characteristics or a propensity score, then compare post-program outcomes with SPSS.

Compute propensity scores—estimated by logistic or probit regression—from observed covariates to estimate enrollment probability, then apply nearest-neighbor matching to form comparable treated and control groups.

Establish impact with psm by matching units, computing outcome differences within each pair, and averaging these differences to obtain overall impact, with optional use of difference-in-differences.

PSM data requirements include pre-treatment data for enrollment criteria and outcome data for treatment and comparison groups to estimate impact. Prefer data from the same survey for participants and non-participants.

Explore the two key assumptions of the PSM method: observable traits and close propensity scores for accurate nearest-neighbor matching.

The lecture explains three limitations of the propensity score matching method: limited generalizability, need for large data to achieve accurate matching, and potential errors from missing key factors.

Supplement the PSM method with difference-in-differences to achieve more reliable impact estimates by addressing observable differences and mitigating unobservable biases.

Predict program participation using enrollment criteria, compute propensity scores with statistical tools, pair treatment units with similar units, compare outcomes of matched pairs, and average differences to estimate program impact.

Explore regression discontinuity design (RDD) to estimate program impact by comparing outcomes just below and above a cutoff, using a threshold and bandwidth to form treatment and control groups.

Construct counterfactuals in RDD by selecting an appropriate bandwidth around the cutoff and forming treatment and control groups from nearby households, balancing bias with data availability to test the impact.

Identify the three core assumptions for regression discontinuity design: no overlapping programs with the same cutoff, no contamination, and a non-manipulative cutoff yielding near-random assignment near the threshold.

Identify data requirements for RDD, including a continuous score to set bandwidth and enough participants within the eligibility criteria, and collect baseline data to determine cutoff points and counterfactuals.

Explore the limitations of the regression discontinuity design in impact measurement, including restricted generalization near the cutoff and the need for large sample sizes to accurately estimate treatment effects.

Identify threats to validity in impact evaluation, and differentiate internal validity from external validity, then learn how to minimize distortions that mislead decisions and reduce program impact.

Identify and mitigate common threats to internal validity, such as dropout and noncompliance, spillover, and external interferences, to reliably attribute outcomes to the intervention.

Identify attrition as the loss of participants before follow-up, biasing results when dropouts differ from completers and threaten comparability; mitigate by oversampling, incentives, tracking, and using weighting or sensitivity analyses.

Non-compliance threatens internal validity by diluting treatment effects when groups are not adhered to. Mitigate with clear instructions, follow-up, monitoring, and intention-to-treat analysis, including complier average causal effect.

Learn how spillover contamination between control and treatment groups threatens internal validity, and apply cluster randomization, track spillover pathways, and statistical adjustments to mitigate its impact.

Identify external interference from programs, policies, or events like elections or disasters that skew outcomes. Document these events and use controls to mitigate effects on intervention and control groups.

Determine appropriate sample size to avoid underpowered or overpowered studies, detect meaningful impact, and balance precision with limited resources in A/B tests and RCTs.

Explore key parameters for sample size calculation in impact evaluation, including effect size, significance level, power, baseline outcome, variance, and standard deviation.

Use the two-sample t-test formula for continuous outcomes to calculate the sample size per group. Compute using z alpha, z beta, standard deviation, and the expected mean difference.

Compute sample size for binary outcomes using a two-proportion z test or chi-squared test, comparing p1 and p2 with an example vaccination rate; cluster adjustments discussed later.

Adjust sample size for cluster design effects by applying d = 1 + (m − 1) icc and n_adjusted = n × design effect, then compute the number of clusters.

Plan for attrition by inflating the sample using the expected response rate (90%), and adjust for cluster effect to determine the final sample size per group for control and treatment.

Combine qualitative and quantitative data through triangulation to strengthen impact evaluation by capturing narratives, perceptions, and lived experiences that explain how and why an intervention works.

Explore qualitative impact evaluation methods, including the most significant change, outcome mapping, outcome harvesting, and appreciative inquiry, and learn how stakeholders collect evidence to improve programs.

Learn the most significant change method: collect stakeholder stories, select the most meaningful ones via a panel, and use them for learning and program improvement in complex contexts.

Use outcome mapping, a qualitative method of evaluation, to track changes in behavior and relationships of boundary partners, monitoring program influence through intentional design, indicators, and progress markers.

Identify and verify what has changed and how the program contributed through outcome harvesting, using interviews, focus groups, and observations, without predefined outcomes.

Describe how appreciative inquiry centers on positives, builds on strengths, and guides discovery, dream, design, and destiny to foster learning, collaboration, and sustaining program improvements.

Choose the right qualitative impact evaluation method by aligning the evaluation question with context, stakeholders, resources, and time, and decide whether to use outcome harvesting to capture unexpected changes.