What are non-parametric tests? When to use them and which to choose

01What is a non-parametric test?

A parameter is a number that describes a whole population, such as its average or its standard deviation. Parametric tests, such as the t-test and ANOVA, are built around those numbers. They assume the outcome follows a particular shape, usually the bell-shaped normal curve, and they test claims about averages.

Non-parametric tests make fewer assumptions about the shape of the data, which is why many are called distribution-free. Most of them work by replacing each value with its rank, its position when all the values are lined up from smallest to largest, and then analysing the ranks.

They are not assumption-free. Observations still need to be independent, the outcome needs at least an order, and to read a result as a difference in medians the groups need distributions of similar shape. What a non-parametric test usually asks is whether one group tends to have higher values than another, and not whether the averages differ.

02How rank-based tests work

Take five people with monthly incomes of 1,200, 1,500, 1,700, 2,000, and 25,000. The average is 6,280, a figure that describes none of them, and the spread is huge. Replace the incomes with ranks and the picture changes.

On the ranks, the person earning 25,000 is simply the top rank, 5, one step above the person earning 2,000. The extreme value has lost its power to distort the result. Two further consequences follow:

• Any change that keeps the order, such as taking logarithms, leaves the ranks unchanged, so the result does not depend on the scale of measurement.
• The size of the gaps is thrown away. A test on ranks cannot tell that one value is twelve times another, only that it is higher.

Once the data are ranked, the tests compare rank totals between groups (Mann-Whitney U and Kruskal-Wallis), the signed ranks of the changes within pairs (Wilcoxon signed-rank), or ranks within each subject (Friedman test). Each guide shows the calculation with a worked example.

03When to use a non-parametric test

Situations that call for one

• The outcome is ordinal. Likert items, grades, severity ratings, and income bands have an order but no reliable distances between values, so averages are not well defined. Ranks are, and this holds at any sample size.
• The sample is small and normality cannot be relied on. With few cases, the central limit theorem gives little protection, and a histogram cannot reliably show the shape. If the values also look skewed or contain outliers, ranks are the safer choice.
• A numeric outcome is strongly skewed or has extreme values. Income, waiting times, and counts with a long tail are typical. If the sample is not large enough for averages to settle down, use ranks.

Situations where a parametric test is the better choice

• The sample is large. With about 30 or more cases per group, more if the skew is strong, averages are close to normal whatever the data look like, and the t-test and ANOVA hold up well.
• The sample is small but the data are clearly symmetric, with no outliers. The parametric test is then more powerful. In large samples a rank-based test is about 95% as efficient when the data really are normal, and in small samples the gap matters more.
• The question is about averages. A rank test does not estimate the difference in means, so if a decision depends on the average, report the average.

The small-sample convention

Many people learn to use a non-parametric test whenever the sample is small. The reasoning is sound: with few cases you cannot check normality, and you cannot count on the central limit theorem. But the rule only works in one direction. Small samples are also where rank-based tests are weakest. With only a few cases, the number of possible rank arrangements is small, so the smallest achievable p-value can sit above 0.05 however large the real difference is.

So sample size is one input to the choice and not the trigger. The type of outcome and the shape of the data matter more. With fewer than about 5 or 6 cases per group, neither kind of test gives much evidence, and it is better to describe the values than to rely on a p-value.

A way to decide

When you are unsure, run both tests. If they agree, your conclusion does not depend on the assumption. If they disagree, look at the data, because an outlier or a skewed distribution is usually the reason. Decide which test you will report before you look at the results, and report both if you ran both.

04Parametric and non-parametric tests compared

05Each parametric test and its rank-based partner

• Independent t-test and Mann-Whitney U, for two separate groups.
• Paired t-test and Wilcoxon signed-rank, for the same people measured twice.
• One-way ANOVA and Kruskal-Wallis, for three or more separate groups.
• Repeated-measures ANOVA and Friedman test, for the same people measured several times.
• Pearson and Spearman correlation, for the relationship between two variables.

AnalyZ Solutions has no rank-based counterpart for the one-sample t-test or the two-way ANOVA. The chi-square test makes no normality assumption either, because it works on counts of categories.

06Checking whether data are roughly normal

• Look at a histogram or boxplot for each group. Skew and outliers are usually plain to see.
• Think about the variable. Income, waiting times, and counts are usually skewed. Bounded scales pile up at the ends. Heights and large-exam scores are roughly symmetric.
• Use formal tests such as Shapiro-Wilk only as a supplement. In small samples they rarely flag a problem when you most need them to, and in large samples they flag differences too small to matter.
• Check the right thing. For a t-test or ANOVA, look at the outcome within each group. For regression, look at the residuals.

07How to find these tests in AnalyZ Solutions

Open Inferential Analysis. The Non-parametric Tests category holds the Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis, and Friedman tests, and Spearman correlation is in the Correlation category. Each guide covers the steps. Use the Descriptive Analysis module first to look at the distributions and counts per group.

The Wilcoxon module asks for at least 5 pairs. As the table above shows, at least 6 pairs are needed before a result can reach 5% significance, so treat results from 5 pairs as untestable.

08Interpreting the result

The figures below are invented to show why the choice matters. They compare the monthly incomes of 8 people who completed a training course with 8 who did not. One person in the second group earns 14,000.

Reading it step by step

1. Compare the means with the medians
   The comparison group has the higher mean (3,600 against 2,938), but the lower median (2,150 against 2,950). When the mean and the median point in opposite directions, an extreme value is usually responsible. The standard deviation of the second group, 4,208, is more than eleven times that of the first.
2. Read the t-test
   The p-value of 0.6641 is far above 0.05, so the t-test finds nothing. The single income of 14,000 has inflated both the average and the spread of its group, which swamps the real gap between the others.
3. Read the rank test
   The rank totals are 91 for the first group and 45 for the second. U is 55 out of a possible 64, so in 86% of all pairs the person who completed the course earns more. The p-value is 0.0148, significant at the 5% level, and the effect size r is 0.72, a large effect. The result does not depend on any single person.
4. Check the cause
   Without the 14,000 income, the t-test gives a p-value of 0.00022 and agrees with the rank test. Before reporting anything, check whether 14,000 is a data entry error. If it is real, say so and report both tests.

In plain words. People who completed the course earned more than those who did not, with a median of 2,950 against 2,150. The difference is large and significant at the 5% level on a rank-based test. A t-test missed it because one very high income in the comparison group distorted the averages. The data cannot show that the course caused the gap, since the groups may have differed before it.

09Common mistakes

Using a rank test only because the sample is small. If the data are clearly symmetric, the t-test is more powerful, and with a handful of cases a rank test may be unable to reach significance at all.

Treating non-parametric as assumption-free. Independence and a meaningful order are still required, and similar shapes are needed for a reading in medians.

Describing the result as a difference in means. A rank test compares ranks. Report medians and mean ranks.

Choosing the test after seeing which one is significant. Decide in advance, or report both.

Relying on a normality test alone. Look at the data and think about the variable.

Reading a non-significant result in a small sample as no difference. It means the data gave too little evidence.