Small groups? Rating scales? Comparing 3+ groups? Meet the Kruskal-Wallis test

01What is the Kruskal-Wallis test?

The Kruskal-Wallis test answers a simple question: when you have three or more groups of people, do they all tend to score alike, or does at least one group tend to score higher or lower than the others?

Suppose delivery times in hours are recorded for 25 deliveries from each of four depots. One depot looks faster and another slower, but every depot also has a few very long deliveries. Are the depots truly different, or is the gap down to which deliveries happened to be sampled? If they differ, which ones? The test gives you a way to tell.

Why not compare averages?

The usual way to compare groups is to compare their averages, and the standard test for that is one-way ANOVA. It works well when the values spread evenly around each group's average in a roughly bell-shaped pattern. It works badly in three common situations.

• Ratings or bands. Satisfaction ratings and income bands are ordinal data, meaning the answers have an order but the gaps between steps may not be equal.
• A few very large values. Delivery times often include a few extremely slow deliveries that drag a group's average up.
• Small groups. With only a few people in a group, its average can jump around from one small sample to the next.

How the test gets around the problem

Instead of the exact values, the test uses their order. It lines everyone from all the groups up from lowest to highest and gives each person a position number, called a rank. If the groups were really the same, each group's members would be spread evenly along the line, so every group would have about the same average position. The test checks whether the groups' average positions differ by more than luck would explain. A very large value is only the last in line, so it cannot drag the result around. For background on this whole family of tests, see What are non-parametric tests?

An example

Suppose delivery times in hours are recorded for 25 deliveries from each of four depots. The times are heavily skewed, because a few deliveries take very long. You want to know three things. Do the depots differ? Which ones? And how large is the difference? The Kruskal-Wallis test answers all three.

What you get out of it

1. Direction
   Which groups tend to have higher or lower values.
2. Size
   How much of the variation in the data is linked to which group a person is in, from 0% to 100%. This is called the effect size.
3. Confidence
   How sure you can be that the groups really differ and it is not luck. This is called the significance level. At the 5% level, differences this large would turn up by luck alone in fewer than 5 samples out of 100.
4. Which groups differ
   Comparisons between pairs of groups, adjusted for how many pairs were compared.

02When to use it

Use it to compare three or more separate groups, meaning each person belongs to only one of them, when at least one of these is true:

• The outcome is a rating or a band, such as satisfaction from 1 to 5 or an income bracket.
• The outcome is a number, but a few very large or very small values would distort the averages.
• The groups are small, so you cannot tell what the data would look like with more people.

It is a common choice for satisfaction ratings, income bands and count-like outcomes such as days absent.

When the values spread evenly around each group's average and the groups are moderate in size, one-way ANOVA is slightly better, because it uses the exact values. For two groups only, use the Mann-Whitney U test. For repeated measurements on the same people, use the Friedman test. For help choosing between this test and ANOVA, see What are non-parametric tests?

03The building blocks: ranks, mean ranks and follow-up comparisons

Three ideas make the test work: giving everyone a position number, comparing the groups' average positions, and following a significant result with comparisons between pairs of groups.

Ranks across all groups

A rank is a position number. Put the values from every group into one list and number them from smallest to largest. If two or more values are equal, they share the average of the positions they occupy. Only the order matters, so the largest value gets the last rank, however large it is.

Mean ranks

A group's mean rank is the average position number of its members. If the groups were equivalent, every group would have about the same mean rank, near the middle of the line. The test asks whether the mean ranks differ by more than luck would allow.

Why a follow-up is needed

A significant result says that at least one group tends to differ from the others. It does not say which. Comparing pairs of groups finds out. Each comparison is adjusted for the number of pairs, because the more comparisons you run, the more likely it is that one of them looks significant by luck alone.

04How the test is calculated

The calculation has four steps. The formulas are there for readers who want to check the arithmetic, and the worked example below follows every step with real numbers.

1. Rank everyone together
   Put the values from all groups into one list and give each a rank, from smallest to largest. Equal values share the average of their positions.
2. Add up the ranks in each group
   The total for each group is its rank sum.
3. Calculate H
   H measures how far the groups' rank sums are from what they would be if the groups were equal. The bigger H is, the bigger the differences.
4. Check H against a reference curve
   The chi-square curve shows how large H tends to be by luck alone. Which curve to use depends on the number of groups: the degrees of freedom, the number that picks the curve, is the number of groups minus 1. The p-value is the chance of seeing an H this large by luck alone if the groups were really the same.

In symbols, with N people in k groups, where group i has nᵢ people and rank sum Rᵢ:

If the groups were equivalent, each group's average rank would be close to the overall average of (N + 1) ÷ 2, and H would be small. Software adds a correction when there are many tied values. The effect size eta² (said "eta squared") is the share of the variation in the ranks that is linked to group membership. It runs from 0 to 1, or 0% to 100%.

When H is significant, the follow-up uses Dunn's procedure, a way of comparing each pair of groups using their mean ranks. For two groups i and j:

The z value says how many standard errors apart the two mean ranks are. The standard error measures how much the difference would vary by luck alone. The adjusted p multiplies each pair's raw p-value by the number of pairs. This is called a Bonferroni adjustment, and it protects against false findings.

A worked example

Five delivery times from each of three depots.

1. Rank all fifteen times together
   The shortest time gets rank 1 and the longest rank 15. All the values differ, so there are no ties.
2. Add the ranks in each group
   The rank sums are 15, 41 and 64, which add to 120, equal to 15 × 16 ÷ 2, so the ranking is consistent.
3. Calculate H
   H = 12 ÷ (15 × 16) × (15² ÷ 5 + 41² ÷ 5 + 64² ÷ 5) - 3 × 16 = 0.05 × 1,200.4 - 48 = 12.02.
4. Find the p-value
   With 3 groups there are 2 degrees of freedom. H = 12.02 gives p = 0.0025, about 0.25%, so the depots differ at the 1% level.
5. Judge the size
   eta² = (12.02 - 3 + 1) ÷ (15 - 3) = 0.835. Depot is linked to about 84% of the variation in ranks, a large effect, because the groups barely overlap.
6. Find which pairs differ
   The standard error of the difference between two mean ranks is √(15 × 16 ÷ 12 × (1/5 + 1/5)) = 2.828. Dividing each difference in mean ranks by it gives the z values in the table below, and the adjusted p-values multiply the raw ones by the 3 pairs.

Only Depot A and Depot C differ clearly after the adjustment, even though the overall test is significant. Depot B sits between them and cannot be separated from either with groups this small.

How it differs from one-way ANOVA

One-way ANOVA is the standard test for comparing the averages of three or more separate groups.

05Assumptions

Every test gives trustworthy results only when some conditions hold. These are called assumptions. For this test they are modest.

• The groups are separate. Each person is in one group only, and one person's answer does not influence another's.
• The outcome has an order. You can say which of two values is higher. Ratings and numbers both qualify.
• Each group has several people. A group with only two or three people carries little weight.
• For a reading as a difference in medians, the groups have a similar shape. If the shapes differ a lot, the test still detects that the groups are different, but you cannot summarise the difference as a shift in the middle value.

06Reading the result

Medians and mean ranks

The descriptive table shows each group's number of people, median and mean rank. The median is the middle value, so half the group scores above it and half below. The mean rank is the average position number of the group's members in the combined line, and it is the quantity the test actually compares. A group with a higher mean rank tends to have higher values than the rest.

The significance level and effect size

The significance level tells you how sure you can be that the groups really differ. "Significant at the 5% level" means that if the groups were truly the same, differences this large would appear by luck in fewer than 5 samples out of 100. The effect size tells you how big the differences are, which is a separate question. It is eta-squared, calculated from the test statistic, and it shows the share of the variation in the ranks that is linked to group membership. Under 1% is negligible, 1% to 6% small, 6% to 14% medium and above 14% large.

Pairwise comparisons after a significant result

A significant overall result says that at least one group differs, but not which. When the overall test is significant, the module compares every pair of groups using Dunn's procedure. It reports each pair's significance after a Bonferroni adjustment, which multiplies the raw p-values by the number of comparisons. A pair flagged as significant is still significant after allowing for how many pairs were compared. Follow-up comparisons like these are often called post-hoc comparisons, meaning "after the fact".

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Non-parametric Tests, then Kruskal-Wallis.

1. Select the numeric variable
   Pick the outcome.
2. Select the grouping variable
   Pick the column that says which group each person belongs to. It needs three or more groups.
3. Apply a subsample filter if needed
   For example, to compare groups within one country office.
4. Click Run
   The output shows the descriptive table, the test result with the effect size, the pairwise table when the overall test is significant, and a plain-language interpretation that names the pairs that differ.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose delivery times in hours were recorded for 25 deliveries from each of four depots. The times are heavily skewed, so a rank-based test is used.

Reading it step by step

1. Read the overall test
   The p-value is below 0.001, or 0.1%, shown by three stars. If the four depots were really the same, differences this large would turn up by luck in fewer than 1 sample in 1,000. So delivery times do differ across the depots.
2. Judge how big the differences are
   Eta-squared is 0.196, so depot accounts for about 20% of the variation in ranks. That is a large effect.
3. Find which depots differ
   Depot D has by far the lowest median (10.6 hours) and the lowest mean rank (28.12). It differs from Depot A (adjusted p 0.0261), Depot B (below 0.001) and Depot C (below 0.001). The other pairs are not significantly different, with adjusted p-values of 1.0000.
4. Trust the adjusted values
   Six pairs were compared, so each p-value was multiplied by six. A pair that is still significant after that adjustment is safe to report.

In plain words

Delivery times differ across the four depots, and the difference is large and significant at the 0.1% level. Depot D is clearly faster than the other three, which are similar to each other. The test shows where the differences are and does not explain them. Staffing, routes or the type of goods handled could all matter.

09Common mistakes

Stopping at the overall test. It says only that some group differs. State which groups differ, using the pairwise results.

Running several Mann-Whitney tests instead. Each extra test raises the chance of finding a difference that is not real.

Describing results as differences in averages. Use medians and mean ranks.

Ignoring unequal spread. If one group is far more spread out than the others, a significant result may reflect the spread as much as the typical value.