How to tell if three or more groups differ

01What is ANOVA?

Analysis of variance, shortened to ANOVA, is a family of tests for whether the average of a numeric outcome differs between groups. The name looks odd, since the aim is to compare means, but the method works by analysing variation. It asks whether the groups differ from each other by more than people differ within each group.

Picture exam scores in three schools. If the school averages sit far apart and the scores inside each school are tightly bunched, the schools clearly differ. If the averages are close and the scores inside each school are widely spread, the schools overlap so much that the gaps could easily be chance. ANOVA turns this picture into a number. It splits the total variation in the outcome into variation between groups, which reflects how far each group average sits from the overall average, and variation within groups, which reflects how much individuals differ from their own group average. The ratio of the two is the F statistic. A large F means the groups differ more than the spread within them would lead you to expect.

The null hypothesis is that all group averages are equal. The test asks how surprising your data would be if that were true, and reports the answer as a significance level.

Why not run several t-tests?

With three groups you could run three t-tests, one for each pair. Each test carries its own chance of a false alarm, and those chances add up. With five groups there are ten pairs, and at the 5% level the chance of at least one false finding is roughly 40%. ANOVA avoids this by asking one question first: is there any difference among the group averages at all? Only if the answer is yes does it make sense to look at which pairs differ.

The ANOVA family

One-way ANOVA has a single grouping variable and is the subject of this guide. Two-way ANOVA has two grouping variables and tests whether their effects depend on each other. Repeated-measures ANOVA is for the same people measured several times. Kruskal-Wallis is the rank-based partner for skewed or ordinal outcomes. All of them rest on the same idea of splitting variation.

A significant ANOVA says that at least one group differs from at least one other. It does not say which, so a follow-up comparison of pairs is part of the routine.

02ANOVA and regression: how they differ

ANOVA and linear regression look like separate tools, and they are taught separately, but they are close relatives. Both are special cases of one general linear model, in which the outcome is a prediction based on the predictors plus unexplained error. A one-way ANOVA is a regression with one categorical predictor. If you code the groups, run a linear regression on the same outcome, and read the overall F test, you get the same F and the same significance level. The regression R-squared equals the ANOVA eta-squared. The regression intercept is the average of the reference group, and each coefficient is the gap between another group and that reference.

What differs is the question each one is built to answer and the output it puts first.

Put simply, ANOVA asks whether groups differ, and regression asks by how much, and with what else held constant. When the only question is whether three or more groups differ on an outcome, either gives the same overall answer, and ANOVA presents it in the way most readers expect. Once you need to adjust for other variables, include a numeric predictor such as age, compare each group with a chosen reference, or predict the outcome, use regression.

03When to use ANOVA

Use one-way ANOVA when all of these hold:

• The outcome is numeric, such as a test score, a measurement, a spending amount, or a scale total.
• There is one categorical grouping variable with three or more groups.
• Each person or case belongs to one group only, so the groups are independent.
• The groups themselves are your main interest, and you want to know whether their averages differ.

Typical examples are test scores across three teaching methods, customer satisfaction across four branches, crop yield under three fertilisers, or recovery time across the arms of a trial.

Choose a different method when your situation changes:

• For exactly two groups, use the independent t-test, which gives the same answer.
• For the same people measured repeatedly, use repeated-measures ANOVA.
• For two grouping variables, use two-way ANOVA.
• For a skewed or ordinal outcome, or very small groups, use Kruskal-Wallis.
• To adjust for other variables or include numeric predictors, use linear regression.
• For a yes or no outcome, use logistic regression, and for counts use Poisson regression.
• For respondents nested in clusters such as schools or clinics, use multilevel regression.

04Assumptions

Independent observations. Each person appears once and is unrelated to the others in the data.

Roughly normal outcomes in each group. ANOVA is fairly tolerant of this when groups are moderately large.

Similar spread across groups. ANOVA is most reliable when group standard deviations are close. As a quick check, compare the standard deviations in the descriptives table. If the largest is more than about twice the smallest, and group sizes differ a lot, be cautious with the result.

05Reading the result

The ANOVA table

The table splits the variation into between-group and within-group sources, each with a sum of squares, degrees of freedom, and mean square. The F statistic and its significance level tell you whether the group averages differ by more than chance would suggest. A significant F means at least one group differs from the others. It does not say which.

Eta-squared, the effect size

Eta-squared is the share of the total variation in the outcome that the grouping variable accounts for. Under 6% is small, 6% to 14% medium, and over 14% large. If school explains 8% of the variation in exam scores, the effect is medium: school matters, but most of the variation is still between individual students.

Pairwise comparisons

When the F test is significant, the follow-up table compares every pair of groups, giving the difference in averages and its significance. This is where you find out which groups differ.

06How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Compare Means, then One-way ANOVA.

1. Select the dependent variable
   Choose the numeric outcome.
2. Select the grouping variable
   Choose the categorical variable that defines the groups. The page shows how many groups it detected.
3. Decide on pairwise comparisons
   The box for pairwise comparisons is ticked by default. They appear only when the overall F test is significant at the 5% level.
4. Apply a subsample filter if needed
   For example, to compare schools among girls only.
5. Click Run
   The output has group descriptives with count, mean, and standard deviation, the ANOVA table with eta-squared, the pairwise table, and a plain-language interpretation naming which groups differ.

You can export as CSV, Excel, or Word, send the finding to Sense Making, or add a bar chart with the group averages to a Data Canvas slide.

07Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose exam scores were collected from 30 students in each of three schools.

Reading it step by step

1. Start with the overall test
   The F test has a p-value of 0.0074 with two stars, so the school averages are not all equal, and the difference is significant at the 1% level. This says at least one school differs. It does not yet say which.
2. Judge the overall size
   Eta-squared is 0.107, so school explains about 11% of the variation in scores. That is a medium effect. Most of the variation is still between individual students.
3. Use the pairwise table to find the differences
   Mean diff is Group 1 minus Group 2, so a negative value means the second group scored higher. School C scored 7.4 points higher than School A (p 0.0023, two stars). School B and School A did not differ significantly (p 0.2500). School C and School B differ by 4.7 points, with a p-value of 0.0477, only just under 0.05.
4. Treat borderline pairs with care
   These pairwise p-values are not adjusted for the number of comparisons. The School A and School C gap is safe to report. The School B and School C gap is borderline and should be described as suggestive.

In plain words

Exam scores differed across the three schools, and the differences were medium in size and significant at the 1% level. School C scored clearly higher than School A. School B fell between them and was not clearly different from either, apart from a borderline gap with School C. The test shows that schools differ, and it does not explain why.

08Common mistakes

Running several t-tests instead. This inflates the chance of false findings. Use ANOVA first, then pairwise follow-ups.

Stopping at the F test. A significant result says something differs. The reader needs to know what.

Including tiny groups. A group of three or four people has an unstable average that can distort comparisons. Consider merging sparse categories where it makes conceptual sense.

Reading eta-squared as causation. The grouping variable may be associated with the outcome through other factors that differ between groups.