Small sample? Rating scales? Compare paired measurements with the Wilcoxon signed-rank test

01What is the Wilcoxon signed-rank test?

The Wilcoxon signed-rank test answers a simple question: when the same people are measured twice, did their scores tend to go up or down?

Suppose 25 patients rate their pain from 0 (none) to 10 (worst) before a treatment and again afterwards. Most ratings look lower the second time. Is that a real change, or could it have happened by chance? The test gives you a way to tell.

Why not use the average change?

The usual way to check a change like this is to work out each person's change and test whether the average change differs from zero. The standard test for that is the paired t-test. It works well when the changes spread evenly around their average in a roughly bell-shaped pattern. It works badly in three common situations.

• Ratings. A pain scale from 0 to 10 gives ordinal data, meaning the answers have an order but the steps may not be equal. A fall from 8 to 6 need not mean the same as a fall from 4 to 2.
• A few very large changes. One patient whose pain fell by nine points can pull the average change a long way.
• Few pairs. With only a handful of people, the average change is unreliable.

How the test gets around the problem

The test works out each person's change, sets aside for a moment whether it was up or down, and ranks the changes by size. The smallest change gets position 1, the next smallest position 2, and so on. Then it puts the direction back, adding up the positions of the increases on one side and the decreases on the other. If the treatment made no difference, the two sides would be about equal. If most of the big changes go the same way, they are not. A very large change is only the last position, so it cannot dominate. For background on this whole family of tests, see What are non-parametric tests?

An example

Suppose 25 patients rate their pain from 0 to 10 before and after a treatment. You want to know three things. Did the ratings tend to change? In which direction? And how consistently did patients move the same way? The Wilcoxon signed-rank test answers all three.

What you get out of it

1. Direction
   Whether scores tended to rise or fall, summarised by the median change. The median is the middle value of the changes, so it shows the typical change per person.
2. Size
   How consistently the changes point the same way, from 0 (ups and downs balance) to 1 (every change goes the same way). This is called the effect size.
3. Confidence
   How sure you can be that the change is real and not luck. This is called the significance level. At the 5% level, a pattern this lopsided would turn up by luck alone in fewer than 5 samples out of 100.

02When to use it

Use it when each person is measured twice, or when people are matched in pairs, and at least one of these is true:

• The outcome is a rating or another measure with an order but unequal steps.
• A few very large or very small changes would distort the average change.
• The sample is small, so you cannot tell what the data would look like with more people.

Examples include a pain or wellbeing rating before and after a treatment, a knowledge test taken by a small pilot group, and a confidence score recorded at the start and end of a workshop.

When the changes spread evenly around their average and the sample is moderate, the paired t-test is the standard choice. For two separate groups, use the Mann-Whitney U test. For three or more measurements on the same people, use the Friedman test. For help choosing between this test and the t-test, see What are non-parametric tests?

03The building blocks: differences, signs and ranks

Three ideas make the test work: looking at the change for each person, ranking the sizes of the changes, and comparing increases with decreases.

The change for each person

For each person, subtract the first measurement from the second. A pain rating of 7 before and 4 after gives a change of -3. A person whose two values are identical has a change of zero, which says nothing about direction, so that person drops out.

Ranks and signs

A rank is a position number. Set the direction aside and line up the sizes of the changes from smallest to largest, numbering them 1, 2, 3 and so on. If two sizes are equal, they share the average of their positions. Then give each position its sign back: plus if the person's score rose, minus if it fell. If the treatment made no difference, the plus and minus positions would balance. If most of the big changes go one way, they do not.

The median change

The median is the middle value, so half the changes are above it and half below. The test reports the median of the changes. It is the typical change per person, and the number to quote when you describe how much scores moved.

04How the test is calculated

The calculation has four steps. The formulas are there for readers who want to check the arithmetic, and the worked example below follows every step with real numbers.

1. Find each change
   Subtract the first measurement from the second for every pair.
2. Drop pairs with no change
   A change of zero says nothing about direction, so those pairs leave the calculation.
3. Rank the sizes
   Ignore the signs and give the remaining changes position numbers from smallest to largest. Equal sizes share the average of their positions.
4. Put the signs back and add up
   W⁺ is the sum of the positions of the increases and W⁻ is the sum of the positions of the decreases. The smaller of the two is the test statistic W.

If the measurements made no difference, increases and decreases would be equally likely. The positions add up to a total of n × (n + 1) ÷ 2, where n is the number of pairs left after the no-change pairs are dropped, and each side would collect about half of it. A W far below that expected half means the changes lean one way.

For larger samples, W is converted into a z value, which says how many standard errors W sits from the value expected if nothing had changed. The standard error measures how much W would vary from sample to sample by luck alone. The p-value then comes from the normal curve, the familiar bell shape. The p-value is the chance of seeing a pattern this lopsided by luck alone if the measurements really made no difference.

The effect size r is the difference between the two sums divided by their total. It runs from 0, when increases and decreases balance, to 1, when every change points the same way.

A worked example

Eight patients rate their pain from 0 to 10 before and after a treatment.

1. Find the changes
   After minus before, they are -6, -2, -4, -1, 0, -3, -5 and -7.
2. Drop the zero
   Pair 5 did not change, so the number of pairs falls from 8 to 7.
3. Rank the sizes
   The sizes 1, 2, 3, 4, 5, 6 and 7 are all different, so they take positions 1 to 7.
4. Add up by sign
   Every change is a decrease, so W⁻ = 28 and W⁺ = 0. The test statistic is W = 0.
5. Find the p-value
   With 7 pairs, the chance that all seven changes point the same way by luck alone is 2 × (1/2)⁷ = 0.0156. That is the p-value, about 1.6%, which is below 5%, so the result is significant at the 5% level. The effect size is 1.0, because every change went the same way.

A tiny sample can still be significant, but only when the pattern is almost perfectly consistent. With 7 pairs, if one of the larger changes (size 3 or more) had gone the other way, the p-value would rise above 0.05.

How it differs from the paired t-test

The paired t-test is the standard test for checking whether the average change between two measurements differs from zero.

05Assumptions

Every test gives trustworthy results only when some conditions hold. These are called assumptions. For this test they are modest.

• The pairs are independent. One person's change does not influence another's.
• The size of a change means something. The test ranks the sizes of the changes, so a change from 2 to 4 should count for more than a change from 2 to 3.
• The changes are roughly symmetric. Around their middle value, the changes should look roughly like a mirror image, with about as much spread on one side as on the other. They do not need a bell shape.

Only complete pairs are included. Anyone missing either measurement is left out, so report how many pairs the test used.

06Reading the result

Medians and the median difference

The output lists the median and the mean for each measurement and for the change between them. The median is the middle value, so half the people score above it and half below. The median difference is the headline number: the typical change per person. The interpretation states its direction and size in the units of your variable.

The significance level

The significance level tells you how sure you can be that the change is real. AnalyZ Solutions reports it as 5%, 1% or 0.1%, and the smaller the percentage, the stronger the evidence. "Significant at the 5% level" means that if the measurements really made no difference, a pattern this lopsided would appear by luck in fewer than 5 samples out of 100. A result that is not significant means the evidence is not strong enough to claim a consistent change. That is not the same as showing that nothing changed.

The effect size

The effect size r tells you how consistently the changes point the same way, which is a separate question from how sure you can be. It runs from 0, when increases and decreases balance, to 1, when every change goes the same way. AnalyZ Solutions labels it negligible below 0.1, small from 0.1 to 0.3, medium from 0.3 to 0.5 and large above 0.5. A high value means most of the changes point the same way.

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Non-parametric Tests, then Wilcoxon Signed-Rank.

1. Select the first variable
   Pick the earlier measurement, such as the before rating.
2. Select the second variable
   Pick the later measurement, such as the after rating.
3. Apply a subsample filter if needed
   For example, to test the change among one group only.
4. Click Run
   The output includes the medians, the median difference, the test statistic, the significance level, the effect size and a plain-language interpretation.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 25 patients rated their pain from 0 to 10 before and after a treatment. Variable 1 is the before rating and Variable 2 is the after rating.

Reading it step by step

1. Read the direction
   The Difference row is the second variable minus the first, so it is after minus before. A median difference of -2.0 means that pain fell by two points for the typical patient, from a median of 6 to a median of 4.
2. Check how sure you can be
   The p-value is below 0.001, or 0.1%, shown by three stars. If the treatment made no difference, a pattern this one-sided would turn up by luck in fewer than 1 sample in 1,000, so chance is a very unlikely explanation.
3. Judge how big the change is
   The effect size r is 0.968, a large effect. It reflects how consistently people moved the same way: almost every patient improved.
4. Check the number of pairs
   The test used 25 pairs, but 3 patients reported no change. Those three carry no information about direction and drop out of the ranking, which leaves 22 pairs doing the work. If many patients had shown no change, the test would have less to work with.

In plain words

Pain ratings fell by a median of 2 points after the treatment, from 6 to 4. The fall was consistent across patients and significant at the 0.1% level, with a large effect size. With no comparison group, the result shows that ratings changed. It cannot show that the treatment caused the change.

09Common mistakes

Using it on separate groups. If the two columns describe different people, use the Mann-Whitney U test.

Ignoring the people with no change. When many people show no change, the test has less to work with. Report how many stayed the same.

Assuming the treatment caused the change. Without a comparison group, other things could explain the change, such as people feeling better with time.

Reporting averages from a rank test. Use medians to describe the results, since the test compares positions in a line and not averages.