Quantitative Analysis/ Effect Analysis
When there is no control group

How to evaluate a programme without a control group

Most programmes are not run as trials, so most evaluations have no control group. This guide shows what you can build in its place, how to choose between the four approaches in AnalyZ Solutions, and how to word the result honestly.

10 min read Effect Analysis Beginner

01What a control group does, and why you may not have one

A control group is a set of people who are like the people in your programme in every way except one: they did not receive it. It is the cleanest way to answer the question every funder eventually asks, which is what would have happened to these people without us?

The strongest version is a randomised control group. People are assigned to the programme or to the control group by chance, so the two groups differ only by luck. Any difference in results afterwards can be put down to the programme.

Most programmes do not have one. The programme may already be running when someone asks for an evaluation. Everyone who is eligible may be getting it, which leaves nobody to leave out. Choosing who gets help by chance may be unacceptable to the community or to your own organisation. Or the budget may not have covered the extra data collection.

That does not mean you cannot say anything credible about impact. It means you need to be careful about what you build in place of the control group, and honest about what that substitute can and cannot show.

02Three shortcuts that look like evaluation but are not

The shortcutWhat it tells youWhy it falls short
Compare participants before and afterHow much the outcome changed.Everything else that changed in the same period is included. The season, prices, other programmes and ordinary growth all get credited to you.
Compare participants with people who did not joinHow the two groups differ today.People who join are usually different from people who do not. They may be better off, more motivated or nearer to the project office.
Ask participants whether the programme helpedWhat people believe about the programme.It is useful evidence about how people experienced it. It does not measure how much of any change the programme caused. People tend to be polite and to remember what helped.

Each of these has a use as supporting evidence. None of them, on its own, is an estimate of the programme's effect.

03What you can build instead: a comparison group

The usual substitute is a comparison group. It plays the same role as a control group, but nobody assigned it by chance. It is found, or built, from data you already have. Because chance did not make it, you have to give reasons for believing it is comparable, and you should say plainly what those reasons are.

Four approaches are built into AnalyZ Solutions. Each one finds its comparison in a different place.

Where the comparison comes fromWhat it looks likeGuide
Similar people who were not reachedHouseholds with the same age, education, income and location as participants, who did not take part. Matching picks the closest ones.Propensity score matching
The trend in a group that did not get the programmeThe same outcome measured before and after for participants and for a comparison group. The comparison group's change shows what time alone did.Difference-in-differences
People just on the other side of an eligibility lineHouseholds that scored just above the poverty cutoff compared with those that scored just below.Regression discontinuity
A fair nudge towards the programmePeople who were invited by lottery or by a random roll-out, compared by whether the invitation moved them to take part.Instrumental variables

04Which one fits your situation

The right approach depends on how your programme was run and what data you have, not on which method you would prefer. Work down these questions and stop at the first yes.

Figure 1. Choosing an approach when there is no control groupA decision chart. Four questions are asked in order. A yes to the first leads to regression discontinuity, to the second difference-in-differences, to the third instrumental variables and to the fourth propensity score matching. If every answer is no, the chart points to the options in section 5.Was eligibility decided by a scorewith a clear cutoff?YesRegression discontinuityNoDo you have measurements before and after,for people who did not get the programme too?YesDifference-in-differencesNoWas access offered by a lottery or randominvitation that not everyone took up?YesInstrumental variablesNoDo you have one survey with participants,non-participants and background details?YesPropensity score matchingNoNone of these fits: see section 5
Figure 1. Ask the questions in order and stop at the first yes

Sometimes two questions get a yes. That is a good position to be in. If you can use both difference-in-differences and matching, run both. When they agree, you can say the finding does not depend on one method's assumptions.

A worked example

A livelihoods project gave start-up grants to 600 women in two districts. Eligibility was decided by a vulnerability score, but the project did not draw a hard line, so cutoff-based methods are out. There was a baseline survey of the 600 grantees and of 450 similar women in a neighbouring district, and an endline survey of both a year later.

The second question gets a yes: measurements before and after, for a group that did not get the programme as well. Difference-in-differences is the starting point. The team would also have background details for everyone, so they could run propensity score matching on the endline data as a second opinion. If both gave a similar effect on monthly income, the report could present the result with more confidence than either alone.

05If none of the approaches fits

Sometimes the data simply are not there. If you are still planning, you have options. If the programme is finished, the choices are narrower, but you still have some.

Before the programme starts

After the programme has ended

06What to write in your report

The strength of the claim you can make depends on the evidence behind it. Match your wording to what you did.

What you didWhat you can say
Before and after, no comparisonOutcomes changed by this amount after the programme. The data cannot say how much of the change the programme caused.
Participants compared with similar non-participants, adjusted for what you measuredParticipants had outcomes this much higher than similar non-participants. Characteristics that were not measured could account for part of the difference.
Difference-in-differences with a believable comparison groupThe programme raised the outcome by this amount relative to the trend in the comparison group, assuming the two groups would have moved in step.
Regression discontinuity or instrumental variablesThe programme raised the outcome by this amount for people near the cutoff, or for people whose participation depended on the invitation.

In every case, give the size of the effect and its confidence interval, state the assumption the method relies on, and say how you checked it. A report that states its assumptions and shows its checks is more convincing than one that claims certainty.

07How to run it in AnalyZ Solutions

Open Effect Analysis from the Quantitative Analysis home screen and choose the method that came out of the questions in section 4. Each guide walks through its form and its results.

  1. Get the data ready in Data View
    Check that the treatment and group variables have two values, that the outcome and background variables have the right type, and that codes such as 99 for refused are marked as missing.
  2. Set the survey design if you have one
    If the data come from a survey with sampling weights, declare the weight, strata and clusters once in Data View.
  3. Choose the method and fill in the form
    Each method asks for the variables it needs.
  4. Read the checks before the effect
    Each method reports checks that show whether its assumptions are believable. Read them first.
When your data come from a survey with sampling weights. All four methods use the survey weight by default when a survey design is set. Estimates are weighted, and the uncertainty comes from the survey design. The guide for each method describes what changes.

08Common mistakes

Assuming any comparison group will do. A group that differs from participants in ways that matter for the outcome gives a misleading answer, however large it is.

Choosing the comparison group after seeing the results. Decide who is in the comparison group, and why, before you look at the outcome.

Using a group that got something similar. If the comparison group received another programme with the same aim, the comparison understates the effect.

Adjusting for things the programme changed. Background characteristics should be measured before the programme or be things it could not change.

Promising a causal result in the proposal. What you can promise depends on the data you will have. Say which approach you plan to use and what it assumes.

Leaving the comparison question until the end. The best time to plan for a comparison group is before the programme starts, when baseline data can still be collected.

Frequently Asked Questions
Is a comparison group the same as a control group?
Not quite. A control group is assigned by chance, so it is comparable by design. A comparison group is chosen or built from existing data, so you have to show that it is comparable. People often use the terms loosely, but the difference matters for how strong a claim you can make.
Can I evaluate a programme with only endline data?
You can, with propensity score matching, if you have background characteristics for participants and non-participants and you believe they cover the main reasons people joined. The result is weaker than one that uses a baseline, and you should say so.
How can I get a baseline after the programme has started?
Sometimes from records, such as registers or earlier surveys that covered the same people. Recall questions in the endline survey can fill gaps. Both are less reliable than a baseline collected at the time, and the report should say how they were obtained.
Will funders accept these methods?
Designs of this kind are widely used in impact evaluation. What funders look for is a clear statement of the method, its assumptions and the checks that were run. It helps to agree the approach with them early.
How big does the sample need to be?
A few hundred observations is a sensible minimum for most methods. Regression discontinuity uses only the people near the cutoff and instrumental variables uses only part of the variation, so both need more.

Estimate what your programme changed

Open Effect Analysis in AnalyZ Solutions. Free to sign up, free to analyze.

Get started
Related Guides