Propensity score matching estimates what a programme did by comparing participants with similar people who did not take part. This guide explains the idea in plain language and shows how to run the analysis, check the match and read the result in AnalyZ Solutions.
Propensity score matching is a way of estimating what a programme did when the people in it were not picked at random. It builds a fair comparison group out of the people who did not take part, choosing those who look most like the participants, and then compares the two groups.
Here is the kind of problem it is made for. A vocational training programme runs in a district, and a survey covers 1,500 households. Of these, 544 took part. Participating households earn about $230 a month and the others about $180, a gap of $51. Is that what the training is worth?
Probably not, because of who took part. The participants were younger, had stayed in school longer and already earned more before the programme started. Those things raise income whether or not anyone attends a course. Some of the gap is the training. Some of it is the kind of household that signs up for training. Matching is a way of separating the two.
| Average for | Participants | Non-participants |
|---|---|---|
| Age of household head (years) | 36.1 | 39.1 |
| Years of schooling | 7.7 | 6.4 |
| Household size | 4.9 | 5.0 |
| Monthly income before the programme ($) | 117 | 104 |
| Distance to market (km) | 8.4 | 9.4 |
| Monthly income now ($) | 230 | 180 |
Use propensity score matching when you have a single dataset, the programme was not assigned at random, and you recorded most of the reasons that made some people join and others stay out. It suits a one-off household or beneficiary survey that includes both participants and non-participants.
Three things need to be in place.
If you have measurements from before and after the programme for both groups, difference-in-differences usually makes better use of that information. If eligibility was decided by a score cutoff, use regression discontinuity. If people were invited at random but not everyone attended, use instrumental variables.
Suppose you want to compare participants with non-participants of the same age. That is easy. Add schooling, then household size, then distance to the market, and soon you are looking for a non-participant who is 36, has seven years of schooling, lives in a household of five and is eight kilometres from the market. Real data rarely contains such a twin.
The propensity score squeezes all those characteristics into one number: the estimated chance that a household with those characteristics takes part in the programme. A household that looks like the typical participant gets a high score, say 0.6. A household that looks like the typical non-participant gets a low one, say 0.2.
Two households with the same score had the same chance of joining. Judging only by what you measured, one of them joined and the other did not by luck. That is what makes them a fair pair. Comparing their incomes tells you something close to what the programme did, because the usual reasons for a difference have been evened out.
Matching only works where the two groups overlap. Figure 1 shows the scores in the example. Participants score higher on average (0.43 against 0.33), but both groups cover most of the range. At the top end a few participants have scores above 0.7 and almost no comparison households do. Those participants have no fair partner, so the matching leaves them out. This is called a lack of common support, and it is one reason the effect is reported only for households that could be matched.
The module works in four steps.
| Technique | How it works | Good to know |
|---|---|---|
| Nearest neighbor matching (1:1, caliper) | Each participant is paired with the one non-participant whose score is closest, as long as the two are within a set distance called the caliper. No comparison household is used twice. | Easy to follow and to check. Participants without a close partner are dropped. |
| Kernel matching (Gaussian) | Each participant is compared with a weighted average of all non-participants, with closer scores counting for more. | Keeps every participant. The bandwidth controls how quickly the weights fade with distance. |
| Inverse probability weighting | No pairs. Each household is weighted by the inverse of the chance of being in its own group, so non-participants who look like participants count for more. | Uses everyone. Very high or very low scores can produce extreme weights. |
| Doubly robust (AIPW) | Combines the weighting with a second model that predicts the outcome. | The estimate holds if either of the two models is right. |
If you are unsure, start with nearest neighbor matching, because you can see the pairs, and then check that kernel matching or the doubly robust estimate gives a similar answer. When different techniques agree, the result does not hang on one technical choice.
Take three participants, each paired with the non-participant whose propensity score is closest. Their monthly incomes in dollars are:
| Pair | Participant | Matched comparison | Difference |
|---|---|---|---|
| 1 | 240 | 205 | 35 |
| 2 | 198 | 171 | 27 |
| 3 | 310 | 281 | 29 |
The average difference is (35 + 27 + 29) ÷ 3 = 30.3. The module does the same for every matched participant, 483 of them in the example.
The module reports one of three effects, chosen under Estimand.
Each effect comes with a standard error, a 95% confidence interval and a p-value. For nearest neighbor and kernel matching, the standard error comes from the spread of the matched differences. For inverse probability weighting and the doubly robust estimate, the module repeats the whole analysis on resampled data, 300 times by default, and uses the spread of those results. This is called a bootstrap.
A value of 0 means the groups are identical on that characteristic. The usual rule is that anything between -0.1 and 0.1 counts as balanced. Rubin's B and R look at the propensity scores themselves. B should be below 25, and R, the ratio of the spreads, should be between 0.5 and 2.
This shows how many participants and comparison households were in the data, the technique and the estimand. For matching techniques it also shows how many participants found a match. If many did not, the effect describes only the ones that did.
A pseudo R-squared for the participation model. It tells you how well your characteristics separate participants from non-participants. It is not a score for the effect. A low value is not a problem for matching, because it means participation was hard to predict from what you measured. A high value, above about 0.4, means the two groups are easy to tell apart, so look at the overlap carefully.
The estimate is the effect in the units of the outcome. The table also shows the standard error, a t statistic, the p-value, the 95% confidence interval and two E-values, which are explained below.
The balance table lists each characteristic with its standardised difference before and after matching. The matching quality line counts how many are within 0.1 and gives the average reduction in the differences.
Matching cannot see what you did not measure, so the module asks how strong a hidden factor would have to be. The E-value is the minimum strength of association that a hidden factor would need with both participation and the outcome to explain the whole effect away. The second E-value does the same for the edge of the confidence interval closest to zero. A larger E-value means the finding is harder to overturn. The bias grid beneath it shows how the estimate would shrink under hypothetical hidden factors of increasing strength.
Open Effect Analysis and choose Propensity Score Matching.
The figures below are invented to show the steps. They come from running the module on a made-up dataset of 1,500 households, of which 544 took part in a vocational training programme and 956 did not. The outcome is monthly household income in dollars. The covariates are the age of the household head, years of schooling, household size, income before the programme and distance to the market. The technique is nearest neighbor matching and the estimand is ATT.
| N treated | N control | N matched | Caliper | Pseudo R² |
|---|---|---|---|---|
| 544 | 956 | 483 | 0.2 | 0.078 |
| Estimand | Estimate | SE | t | p-value | 95% CI lower | 95% CI upper | E-value | E-value (CI) |
|---|---|---|---|---|---|---|---|---|
| ATT | 33.40 | 3.50 | 9.54 | < 0.001 | 26.54 | 40.26 | 4.32 | 3.48 |
| Covariate | SMD before matching | SMD after matching |
|---|---|---|
| Age of household head | -0.334 | -0.013 |
| Years of schooling | 0.472 | -0.014 |
| Household size | -0.018 | -0.015 |
| Income before the programme | 0.240 | 0.001 |
| Distance to market | -0.210 | -0.014 |
| Rubin's B before | Rubin's B after | Rubin's R before | Rubin's R after |
|---|---|---|---|
| 68.05 | 0.31 | 1.04 | 1.01 |
Households that joined the vocational training earned about $33 more per month than similar households that did not, once age, schooling, household size, earlier income and distance to the market were taken into account (95% confidence interval $27 to $40). A direct comparison of participants and non-participants overstated the gain by about $17. Because matching adjusts only for measured characteristics, this is strong evidence of a positive effect and not proof of one.
Matching on things the programme changed. If the training raised income, then matching on income after the programme removes part of the effect you are trying to measure. Use only characteristics that were fixed or measured before the programme.
Leaving out an important reason for joining. If households with a link to the programme staff joined more often and you did not record it, matching cannot fix that. Think about why people joined before you choose the covariates.
Reading the p-value without reading the balance table. A significant effect on a poorly matched comparison group tells you very little.
Not saying how many participants were left out. If a large share finds no partner, the result describes a narrower group than you may intend. Report the numbers.
Treating the propensity model as a prediction exercise. The aim is balance, not a model that predicts participation well. Covariates that predict participation but have nothing to do with the outcome can make matching harder without improving the answer.
Treating a matched estimate as proof of cause. It is a careful comparison of similar households. It is only as good as the list of characteristics behind it.
Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.
Get started