Difference-in-differences compares how an outcome changed over time in a programme group and in a comparison group. This guide explains the logic in plain language, the one assumption it depends on, and how to run and read it in AnalyZ Solutions.
Difference-in-differences compares how an outcome changed over time in a group that received a programme with how it changed in a group that did not. The programme's effect is the extra change in the programme group. The name describes the arithmetic: you take a difference over time in each group, and then take the difference between those two differences.
Take a school feeding programme. Attendance, measured as the percentage of school days a pupil attended in the month, was recorded for 900 pupils at baseline and again at endline. About half of them attend schools that received the programme, and the rest attend similar schools that did not.
| Average attendance (%) | Baseline | Endline | Change |
|---|---|---|---|
| Programme schools | 66.5 | 76.7 | +10.2 |
| Comparison schools | 71.2 | 74.0 | +2.8 |
Attendance rose 10.2 points in the programme schools. Is that the effect of feeding? Not entirely. It also rose 2.8 points in the comparison schools, where nothing was provided. Something else was pushing attendance up everywhere, perhaps better weather at endline or a bad month at baseline. Only the extra 7.4 points can be put down to the programme.
A direct comparison at endline would also mislead. The programme schools started 4.7 points lower than the comparison schools, so they have more ground to make up. Difference-in-differences avoids both traps. It does not care where each group started. It looks only at how much each group moved.
Use difference-in-differences when you measured the same outcome at two points in time, once before the programme and once after, for both a programme group and a comparison group. The programme should have started between the two rounds, and the comparison group should not have received it.
The two rounds can follow the same people (often called panel data) or can be fresh samples drawn from the same schools or villages (repeated cross-sections). Both work.
It is not the right tool in a few situations. With only one round of data, propensity score matching is the usual choice. If eligibility depended on a score cutoff, regression discontinuity is stronger. And if the programme started before your first round, the baseline is no longer a clean before.
The central idea is simple. If the programme schools had not received the programme, their attendance would probably still have moved. The comparison schools show how much. Difference-in-differences takes the comparison group's change and treats it as the change the programme group would have had anyway.
In Figure 1 the dotted line starts at the programme schools' baseline (66.5) and rises by the comparison group's change (2.8). It ends at 69.3. The programme schools actually reached 76.7. The gap between the two, 7.4 points, is the estimate.
Everything rests on one assumption, called parallel trends: without the programme, the two groups would have changed by the same amount. The groups do not need to start at the same level. They need to be moving in step.
This cannot be proved, and with only two rounds of data it cannot be tested either. What you can do is ask whether it is believable. Were the two sets of schools similar in location, size and the kind of families they serve? Did anything else happen during the period to one group and not the other, such as a new road, a conflict, a teacher strike or a different programme? If you have earlier rounds, plot the trends before the programme and check that the lines ran side by side.
Work out the average outcome in each of the four cells, then do two subtractions.
In the example: (76.70 - 66.51) - (74.01 - 71.19) = 10.19 - 2.82 = 7.37.
The module gets the same answer with a regression, which also lets you adjust for other characteristics and gives a standard error. The regression has three main terms.
| Term | What it measures |
|---|---|
| Group (b1) | How different the programme group was from the comparison group at baseline. |
| Round (b2) | How much the comparison group changed between baseline and endline. This is the background trend. |
| Group × Round (b3) | The extra change in the programme group. This is the difference-in-differences effect. |
Group takes the value 1 for the programme group and 0 for the comparison group. Round takes 1 for the endline and 0 for the baseline. Group × Round is 1 only for programme-group observations at endline, which is why its coefficient captures the programme.
The module reports robust standard errors, known as HC1. They allow the outcome to vary by different amounts for different pupils, which is closer to real data than assuming one common spread. They do not allow for the fact that pupils in the same school tend to resemble each other. If you sampled whole schools, treat the confidence interval as somewhat too narrow, or set up the survey design with school as the cluster (see the note in section 6).
If the outcome is binary, such as whether a child attended school at all, the module also fits a logistic regression. The interaction coefficient from that model is on the log-odds scale, and it is not the effect on the probability of the outcome. So the module reports a separate estimate called the average marginal effect, which is the difference-in-differences in probability. Use that one in your report.
The number of observations, the two groups and the two rounds as the module understands them, the number of covariates and the type of standard error.
The four average outcomes, with the number of observations in each. This is the table you can check by hand.
One row for each term, with the coefficient, robust standard error, t statistic, p-value and 95% confidence interval. The row labelled Group × Round (DiD) is the effect. The intercept and the covariate rows are there for completeness and are usually not what you report.
R-squared and adjusted R-squared tell you how much of the variation between individual pupils the model explains. The F test checks that the terms together explain something. Neither says anything about whether the programme effect is credible. A difference-in-differences with a low R-squared can be perfectly sound.
Open Effect Analysis and choose Difference-in-Differences.
The figures below are invented to show the steps, and they come from running the module on a made-up dataset. 1,800 observations come from 900 pupils measured at baseline and endline. The outcome is the percentage of school days attended in the month, the group variable separates programme from comparison schools, and the covariate is the age of the pupil.
| N observations | Group A | Group B | Round A | Round B | Covariates |
|---|---|---|---|---|---|
| 1,800 | Comparison | Programme | Baseline | Endline | 1 |
| Group | Round | Mean | N |
|---|---|---|---|
| Comparison | Baseline | 71.19 | 429 |
| Comparison | Endline | 74.01 | 429 |
| Programme | Baseline | 66.51 | 471 |
| Programme | Endline | 76.70 | 471 |
| Term | Coefficient | SE | p-value |
|---|---|---|---|
| Intercept | 76.582 | 1.492 | < 0.001 |
| Group | -4.730 | 0.590 | < 0.001 |
| Round | 2.824 | 0.577 | < 0.001 |
| Group × Round (DiD) | 7.368 | 0.818 | < 0.001 |
| age | -0.513 | 0.135 | < 0.001 |
| R² | Adjusted R² | F-statistic | F p-value |
|---|---|---|---|
| 0.170 | 0.168 | 92.0 | < 0.001 |
Attendance rose by 7.4 percentage points more in the schools that received the feeding programme than in comparable schools that did not (95% confidence interval 5.8 to 9.0). The estimate assumes that, without the programme, attendance in the two sets of schools would have followed the same trend.
Taking parallel trends for granted. It is the whole basis of the method. State it, say why you believe it, and use earlier rounds to check it if you have them.
Ignoring other events. A road, a conflict or a second programme that reached only one group in the same period will be counted as the programme effect.
Comparing different kinds of people across rounds. If the endline sample covers a different mix of pupils from the baseline, such as only those who stayed in school, the change reflects who was measured as well as what happened to them.
Using a comparison group the programme touched. Spillovers make the comparison group look better than it would have been, which shrinks the estimate.
Starting with a ceiling. If the programme group already attends 95% of days at baseline, it cannot rise as much as a group that starts at 65%. The groups cannot follow parallel trends near a limit.
Reading a short confidence interval as certainty. Pupils in the same school resemble each other, so the true uncertainty is larger than a standard error that treats pupils as independent suggests.
Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.
Get started