Small sample? Rating scales? Comparing 3+ repeated measurements? Meet the Friedman test

01What is the Friedman test?

The Friedman test answers a simple question: when the same people are measured under three or more conditions, do they tend to score some conditions higher than others?

Suppose 30 respondents each rate three kinds of service, a home visit, a phone call and a clinic visit, from 1 to 7. Home visits seem to get the highest ratings. Is that a real pattern, or could it be chance? The test gives you a way to tell.

Why not compare averages?

The usual way to compare repeated measurements is to compare their averages, and the standard test for that is repeated-measures ANOVA. It works well when the scores spread evenly around the average in a roughly bell-shaped pattern, and when the differences between each pair of conditions vary by about the same amount from person to person. It works badly in three common situations.

• Ratings. A scale from 1 to 7 gives ordinal data, meaning the answers have an order but the gaps between steps may not be equal.
• A few very large or very small scores. They can drag an average a long way.
• Small samples. With only a few people, an average can jump around from one small sample to the next.

How the test gets around the problem

Instead of the exact scores, the test uses order within each person. For each person, it ranks their own scores across the conditions, which means giving each condition a position number: the one they rated lowest gets position 1, the next lowest position 2, and so on. It then adds up the positions each condition received across everyone. If the conditions were rated equally, the totals would be about the same. If one condition tends to be rated highest, its total pulls ahead. Because each person is ranked against themselves, a generous rater and a harsh rater count equally. For background on this whole family of tests, see What are non-parametric tests?

An example

Suppose 30 respondents each rate three service types on a scale from 1 to 7. You want to know three things. Are the services rated differently? Which ones? And do respondents agree about the order? The Friedman test answers all three.

What you get out of it

1. Direction
   Which conditions tend to be rated higher or lower.
2. Size
   How consistently people ranked the conditions in the same order, from 0 (no agreement) to 1 (everyone ranked them identically). This is called Kendall's W, and it serves as the effect size.
3. Confidence
   How sure you can be that the conditions really differ and it is not luck. This is called the significance level. At the 5% level, a pattern this strong would turn up by luck alone in fewer than 5 samples out of 100.
4. Which conditions differ
   Comparisons between pairs of conditions, adjusted for how many pairs were compared.

02When to use it

Use it when each person has been measured under three or more conditions, or at three or more points in time, and at least one of these is true:

• The outcome is a rating or another measure with an order but unequal steps.
• The scores are numbers, but a few very large or very small ones would distort the averages.
• The sample is small, so you cannot tell what the data would look like with more people.

Examples include satisfaction with three service types rated by the same respondents, symptom scores at three follow-up visits, and preference ratings given to four versions of a leaflet.

For two measurements on the same people, use the Wilcoxon signed-rank test. For a roughly bell-shaped outcome with an adequate sample, repeated-measures ANOVA is stronger. For separate groups, use the Kruskal-Wallis test. Unlike repeated-measures ANOVA, the Friedman test does not need the differences between pairs of conditions to vary evenly, a condition called sphericity. For help choosing between this test and ANOVA, see What are non-parametric tests?

03The building blocks: ranking within each person and agreement

Three ideas make the test work: each person rates every condition, each person's own scores are ranked, and the rank totals show whether the conditions differ.

Repeated measurements

The same people rate every condition, so each person supplies a set of scores that belong together. The data are arranged with one row per person and one column per condition.

Ranking within each person

A rank is a position number. For each person, rank their own scores from lowest to highest across the conditions. Then add up the ranks each condition receives across all people. Because each person is ranked separately, a generous rater and a harsh rater contribute equally, so differences in general scoring level between people drop out of the comparison.

Agreement

If the conditions were rated equally, each would collect about the same rank total. If people tend to rank the conditions in the same order, the totals pull apart. Kendall's W describes how consistently people agreed on the order, from 0 for no agreement to 1 for complete agreement.

04How the test is calculated

The calculation has four steps. The formulas are there for readers who want to check the arithmetic, and the worked example below follows every step with real numbers.

1. Rank within each person
   For each person, give their own scores position numbers from 1 (lowest) to k (highest), where k is the number of conditions. Equal scores share the average of their positions.
2. Add up the ranks for each condition
   The total for each condition is its rank sum, taken across all people.
3. Calculate chi-square
   This statistic measures how far the rank sums are from what they would be if every condition were rated equally. The bigger it is, the bigger the differences.
4. Check it against a reference curve
   The chi-square curve shows how large the statistic tends to be by luck alone. Which curve to use depends on the number of conditions: the degrees of freedom, the number that picks the curve, is k - 1. The p-value is the chance of seeing a statistic this large by luck alone if the conditions were really rated the same.

In symbols, with n respondents, k conditions and rank sums Rⱼ across the respondents:

Software applies a correction when there are tied scores. Kendall's W rescales the statistic to run from 0, when respondents rank the conditions in unrelated orders, to 1, when they all rank them identically.

A worked example

Six respondents rate three services. Each respondent's scores are ranked across the services.

1. Rank within each row
   In respondent 1, the lowest score is 5.2, so it takes rank 1, then 6.1 takes rank 2 and 7.3 takes rank 3. In respondent 3 the order differs, with 5.5 lowest.
2. Add up the ranks by condition
   The rank sums are 7, 11 and 18. They add to 36, which equals 6 × 3 × 4 ÷ 2, so the ranking is consistent.
3. Calculate chi-square
   12 ÷ (6 × 3 × 4) × (7² + 11² + 18²) - 3 × 6 × 4 = 12 ÷ 72 × 494 - 72 = 10.33.
4. Find the p-value
   With 3 services there are 2 degrees of freedom. A chi-square of 10.33 gives p = 0.0057, about 0.6%, so the result is significant at the 1% level.
5. Judge the agreement
   Kendall's W = 10.33 ÷ (6 × 2) = 0.861. Five of the six respondents ranked the services in exactly the same order, so the agreement is strong.
6. Follow up
   AnalyZ Solutions then compares each pair of conditions with a Wilcoxon signed-rank test, which looks at each person's change between two conditions, and adjusts the p-values for the number of pairs.

How it differs from repeated-measures ANOVA

Repeated-measures ANOVA is the standard test for comparing the average scores of the same people under three or more conditions.

05Assumptions

Every test gives trustworthy results only when some conditions hold. These are called assumptions. For this test they are modest.

• The people are independent of each other. One person's ratings do not influence another's.
• The measurements within a person are comparable. The conditions are rated on the same scale.
• The outcome has an order. You can say which of two scores is higher.

Only people with a value for every selected condition are included, so check how many complete cases remain.

06Reading the result

The descriptive table

The descriptive table lists, for each condition, the number of people, the median and the mean. The median is the middle value, so half the people score above it and half below. The mean is the ordinary average.

The test result

The output shows the chi-square statistic, the degrees of freedom, the significance level and the number of people. A significant result says that the conditions are not all rated equally, and that at least one differs from another. "Significant at the 5% level" means that if the conditions were truly rated the same, a pattern this strong would appear by luck in fewer than 5 samples out of 100.

Kendall's W, the effect size

Kendall's W runs from 0 to 1 and shows how consistently people ranked the conditions in the same order. A value near 1 means nearly everyone agreed on which condition ranked highest and lowest. A value near 0 means the rankings were scattered. AnalyZ Solutions labels W below 0.1 as weak, 0.1 to 0.3 as moderate, and above 0.3 as strong agreement.

Pairwise follow-ups

When the overall test is significant, the module compares every pair of conditions using Wilcoxon signed-rank tests, with a Bonferroni adjustment. The adjustment multiplies each p-value by the number of comparisons, because the more pairs you compare, the more likely it is that one looks significant by luck alone. The p-values you see already include it.

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Non-parametric Tests, then Friedman Test.

1. Arrange your data in wide format
   Each row is one person, and each condition or time point is its own column.
2. Select three or more variables
   Pick the columns that hold the repeated measurements.
3. Apply a subsample filter if needed
   For example, to analyse one site only.
4. Click Run
   The output shows the descriptive table for each condition, the test result with Kendall's W, the pairwise table when the overall test is significant, and a plain-language interpretation.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 30 respondents each rated three service types from 1 to 7.

Reading it step by step

1. Read the overall test
   The p-value is 0.0040, or 0.4%, shown by two stars. If the three services were really rated the same, a pattern this strong would turn up by luck in about 4 samples out of 1,000. So the ratings differ across the services, and the difference is significant at the 1% level.
2. Judge the agreement
   Kendall's W is 0.1843, a moderate level of agreement. Respondents tended to rank the services in a similar order, though not uniformly.
3. Find the pairs
   Home visit and clinic visit differ significantly after adjustment (p 0.0082, two stars), and home visits are rated higher (mean 5.07 against 4.27). The other two pairs are not significantly different.
4. Do not over-read the medians
   Two of the three medians (the middle values) are both 5. When medians tie, the means and the pairwise results say more about which service is rated higher.

In plain words

Respondents rated home visits higher than clinic visits, and the difference is significant at the 1% level after adjusting for multiple comparisons. Phone calls fell between the two and were not clearly different from either. The agreement across respondents was moderate.

09Common mistakes

Using data with one row per rating. The module needs one column per condition and one row per person.

Treating it as a test for separate groups. The same people must appear in every condition.

Ignoring missing data. A person missing one condition is dropped completely, so many missing values shrink the sample quickly.

Confusing agreement with size of difference. Kendall's W shows how consistently people ranked the conditions, and not how far apart the conditions are in the raw scores.