Small sample? Rating scales? Compare two separate groups with the Mann-Whitney U test

01What is the Mann-Whitney U test?

The Mann-Whitney U test answers a simple question: when you have two groups of people, does one group tend to score higher than the other?

Suppose you ask customers who used your online service, and customers who did not, to rate it from 1 (very poor) to 5 (excellent). The first group's ratings look a little higher. Is that a real difference, or did it happen by chance because of who happened to answer? The test gives you a way to tell.

Why not compare averages?

The usual way to compare two groups is to compare their averages, and the standard test for that is the independent t-test. It works well when the values spread evenly around the average in a roughly bell-shaped pattern. It works badly in three common situations.

• Ratings. A scale from 1 to 5 gives ordinal data, meaning the answers have an order but the gaps between steps may not be equal. The gap between a 2 and a 3 need not match the gap between a 4 and a 5, so an average of ratings can mislead.
• A few very large values. Income and waiting times often have a handful of huge values that drag the average up.
• Small samples. With only a few people, an average can jump around from one small sample to the next.

How the test gets around the problem

Instead of the exact values, the test uses their order. It lines everyone from both groups up from lowest to highest and gives each person a position number, called a rank. Then it checks whether the people in one group tend to sit higher in the line. A very large value is only the last in line, so it cannot drag the result around. For background on this whole family of tests, see What are non-parametric tests?

An example

Suppose 85 respondents rate a service from 1 to 5, and they fall into two groups, such as customers who used the online service and customers who did not. You want to know three things. Do the two groups rate the service differently? Which group rates it higher? And how big is the difference? The Mann-Whitney U test answers all three.

What you get out of it

1. Direction
   Which group tends to score higher.
2. Size
   How big the difference is, on a scale from 0 (no difference) to 1 (everyone in one group scores higher than everyone in the other). This is called the effect size.
3. Confidence
   How sure you can be that the difference is real and not luck. This is called the significance level. At the 5% level, a gap this big would turn up by luck alone in fewer than 5 samples out of 100.

02When to use it

Use it to compare two separate groups, meaning each person belongs to only one of them, when at least one of these is true:

• The outcome is a rating or a ranking, such as satisfaction from 1 to 5.
• The outcome is a number, but a few very large or very small values would distort the average, as with income or waiting times.
• The sample is small, so you cannot tell what the data would look like with more people.

Typical examples are satisfaction ratings of men and women, incomes in two cities, and symptom scores in a treatment group and a comparison group in a small pilot.

When the values spread evenly around their average and the groups are large, the independent t-test is slightly better, because it uses the exact values. For three or more groups, use the Kruskal-Wallis test. When the same people are measured twice, use the Wilcoxon signed-rank test. For help choosing between this test and the t-test, see What are non-parametric tests?

03The building blocks: ranks and what the test compares

Two ideas make the test work: turning values into position numbers, and asking whether one group tends to sit higher than the other.

Ranks

A rank is a position number. Line all the values up from smallest to largest and number them 1, 2, 3 and so on. If two or more values are equal, they share the average of the positions they occupy. The ratings 2, 3, 3, 4, for example, get the ranks 1, 2.5, 2.5 and 4, because the two 3s occupy positions 2 and 3, so each gets 2.5. Only the order matters. The largest value gets the last rank, however large it is.

What the test compares

The test asks whether people in one group tend to have higher values than people in the other. When the two groups have a similar shape, this comes down to a difference in medians. The median is the middle value, so half the group scores above it and half below. AnalyZ Solutions reports each group's median because it describes the gap simply.

The test itself uses every position in the line, including those far from the middle. If the groups differ a lot in how spread out they are, the test can find a difference even when the medians match, so always look at the descriptive table as well.

04How the test is calculated

The calculation has four steps. The formulas are there for readers who want to check the arithmetic, and the worked example below follows every step with real numbers.

1. Rank everyone together
   Put the values from both groups into one list and give each a rank, from smallest to largest. Equal values share the average of their positions.
2. Add up the ranks in each group
   Call the totals R₁ for the first group and R₂ for the second. A group whose members sit higher in the line has a bigger total.
3. Convert the total into U
   U counts how many times a person from the first group beats a person from the second, across every possible pairing. A tie counts as half a win.
4. Judge how far U is from "no difference"
   If the groups were equal, each group would win half of all the pairings. The further U sits from that, the stronger the evidence of a real difference.

In symbols, where n₁ and n₂ are the number of people in each group:

U₁ is the number of wins for the first group and U₂ the number for the second. They always add up to n₁ × n₂, the total number of pairings. AnalyZ Solutions reports U for the first group listed.

For larger samples, U is converted into a z value, which says how many standard errors U sits from its no-difference value of n₁ × n₂ ÷ 2. The standard error measures how much U would vary from sample to sample by luck alone. The p-value then comes from the normal curve, the familiar bell shape. The p-value is the chance of seeing a gap this big by luck alone if the groups were really the same. The effect size r compares the share of pairings the first group wins with the share the second group wins.

The effect size r runs from 0, when each group wins half the pairings, to 1, when one group wins them all.

A worked example

Two groups of six people rate a service from 1 to 5. Group A gives 4, 5, 3, 4, 5, 4 and Group B gives 2, 3, 3, 4, 2, 3.

1. Rank all twelve ratings
   The two 2s occupy positions 1 and 2, so each gets rank 1.5. The four 3s occupy positions 3 to 6, so each gets 4.5. The four 4s occupy positions 7 to 10, so each gets 8.5. The two 5s occupy positions 11 and 12, so each gets 11.5.
2. Add up the ranks by group
   Group A's ranks add up to 53 and Group B's to 25. Together they make 78, which equals 12 × 13 ÷ 2, so the ranking is consistent.
3. Convert to U
   U₁ = 53 - 6 × 7 ÷ 2 = 32, and U₂ = 36 - 32 = 4.
4. Read U in plain terms
   There are 36 possible pairings, one person from each group. In 32 of them, counting ties as half, the Group A person gave the higher rating. That is 89% of the pairings. A value of 18 would mean that neither group has the upper hand.
5. Work out how strong the evidence is
   z = (32 - 18) ÷ the standard error = 2.33. The effect size is r = |1 - 2 × 32 ÷ 36| = 0.78, which is the 89% of pairings won by Group A minus the 11% won by Group B. The software reports a p-value of 0.0244. At about 2.4% it is below 5%, so the result is significant at the 5% level. The software makes a small adjustment for the fact that U can only take certain whole values, which is why this is a little larger than the 0.0196 that z alone would give.

How it differs from the independent t-test

The independent t-test is the standard test for comparing the averages of two separate groups.

05Assumptions

Every test gives trustworthy results only when some conditions hold. These are called assumptions. For this test they are modest.

• The groups are separate. Each person is in one group only, and one person's answer does not influence another's.
• The outcome has an order. You can say which of two values is higher. Ratings and numbers both qualify.
• The groups have a similar shape. The test does not need a bell shape. But if one group is bunched tightly and the other is spread widely, a significant result may reflect the spread and not the typical value.

06Reading the result

Medians and mean ranks

The descriptive table lists, for each group, the number of people, the median and the mean rank. The median is the middle value. The mean rank is the average position number of the group's members in the combined line, so a higher mean rank means the group's people generally sit higher. When two groups share a median, which often happens with short rating scales, the mean ranks still show which group tends to score higher.

The significance level

The significance level tells you how sure you can be that the difference is real. AnalyZ Solutions reports it as 5%, 1% or 0.1%, and the smaller the percentage, the stronger the evidence. "Significant at the 5% level" means that if the groups were truly the same, a gap this large would appear by luck in fewer than 5 samples out of 100. A result that is not significant means the evidence is not strong enough to say the groups differ. That is not the same as showing they are equal, especially when there are few people.

The effect size

The effect size tells you how big the difference is, which is a separate question from how sure you can be. For this test it is called r and runs from 0 to 1. A value of 0 means neither group tends to score higher. A value of 1 means every person in one group scores higher than every person in the other. AnalyZ Solutions labels it negligible below 0.1, small from 0.1 to 0.3, medium from 0.3 to 0.5 and large above 0.5. With a big sample even a tiny difference can be significant, so check the size as well as the significance.

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Non-parametric Tests, then Mann-Whitney U.

1. Select the numeric variable
   Pick the outcome, such as a satisfaction rating or income.
2. Select the grouping variable
   Pick the column that says which group each person belongs to. It must have exactly two groups.
3. Apply a subsample filter if needed
   For example, to compare the two groups within one region only.
4. Click Run
   The output shows the descriptive table, the U value, the p-value, the effect size and a plain-language interpretation.

You can export the tables as CSV, Excel or Word, send them to Sense Making, or add a chart to a Data Canvas slide.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 85 respondents rated a service from 1 to 5, and you compare two groups.

Reading it step by step

1. Compare the medians and mean ranks
   Group A has a median rating of 4 and Group B a median of 3, so the typical person in Group A rated the service one point higher. The mean ranks point the same way: 48.51 for Group A and 36.80 for Group B. A higher mean rank means the group's people sit higher in the combined line.
2. Check how sure you can be
   The p-value is 0.0232, or 2.3%. The single star means the result is significant at the 5% level. If the groups were really the same, a gap this large would turn up by luck in about 2 samples out of 100.
3. Judge how big the difference is
   The effect size r is 0.276, a small effect. In plain terms, Group A wins about 64% of the one-to-one pairings and Group B about 36%, a gap of 28 percentage points. The groups differ, but their ratings overlap a lot.
4. Describe it correctly
   This test does not compare averages. It shows that people in Group A tend to give higher ratings than people in Group B, and the medians are the simplest way to say so.

In plain words

People in Group A rated the service higher than people in Group B, with a median rating of 4 against 3. The difference is small but unlikely to be luck, because it is significant at the 5% level. A larger sample would show whether it holds.

09Common mistakes

Calling it a test of averages. It compares positions in the line, so describe the result with medians.

Using it on paired data. If the same people appear in both groups, for example before and after, use the Wilcoxon signed-rank test.

Using it on more than two groups. Running several two-group tests raises the chance of finding a difference that is not real. Use the Kruskal-Wallis test instead.

Ignoring group sizes. With very small groups the test can miss a real difference, so "no significant difference" may simply mean too few people.