Inferential analysis draws conclusions about a population from a sample. This guide builds from the basic ideas of hypotheses, significance, and effect size up to adjustment, clustering, and model choice, and gives two decision trees for picking a test.
Descriptive statistics summarise the data you have. Inferential statistics ask what that data can tell you about people, places, or events you did not measure. A survey of 800 residents cannot include every resident of a city, yet its results are often used to judge what is probably true for the whole city.
The difficulty is that any sample is only one of many that could have been drawn. A different 800 residents would give a slightly different average. Inferential methods measure how much results wobble from sample to sample, and use that to decide which patterns are large and consistent enough to be believed, and which could be produced by chance alone.
The population is the whole group you want to understand. The sample is the part of it you observe. A number calculated from the sample, such as an average, is a statistic. The matching unknown number in the population is a parameter. The gap between the two, caused by observing only part of the population, is sampling error. It shrinks as samples get larger, and it is smallest when the sample is drawn at random.
Random sampling matters because it is what justifies the whole exercise. With a convenience or purposive sample, the tests still describe patterns in your data, but generalising beyond it needs an argument about why your respondents resemble the wider group.
The standard error measures how much a statistic would vary from sample to sample. For an average, it equals the standard deviation divided by the square root of the sample size, which is why quadrupling the sample only halves the error.
A confidence interval turns the standard error into a range. A 95% interval around an average of 4.2 running from 4.0 to 4.4 says that, if the study were repeated many times, about 95% of intervals built this way would contain the true value. In practice it is read as a plausible range for the unknown truth. Wide intervals mean imprecise estimates, whatever their significance.
A hypothesis test starts from a null hypothesis, the plain position that nothing is going on: two groups have the same average, two variables are unrelated, a treatment made no difference. The alternative hypothesis says that something is going on. The test asks how surprising the data would be if the null hypothesis were true.
The answer is the p-value, the probability of seeing a result at least as extreme as yours if the null hypothesis were true. A small value means the data would be hard to explain by chance. Researchers set a threshold in advance, called the significance level, and 5% is the usual choice. A p-value below 5% is called significant at the 5% level, and stricter levels of 1% and 0.1% are used for stronger evidence.
A p-value is not the probability that the null hypothesis is true, and it is not the probability that the result will replicate. It describes the data under one assumption and nothing more.
A Type I error is a false alarm: concluding there is an effect when there is none. The significance level is the risk of this you accept. A Type II error is a missed finding: failing to detect an effect that is really there.
Power is the probability of detecting a real effect of a given size, and it equals one minus the chance of a Type II error. Studies are usually designed for power of 80% or more. Power rises with sample size, with larger true effects, and with less variable outcomes. This is why a non-significant result in a small sample should be read as "not enough evidence" and never as "no effect".
With enough data, a trivial difference becomes significant. With too little, a large one may not. Significance therefore says how confident you can be that an effect exists, and says nothing about how big it is. An effect size measures magnitude independent of sample size.
Cohen's d expresses a difference between averages in standard deviations, with 0.2, 0.5, and 0.8 conventionally called small, medium, and large. Eta-squared gives the share of variation explained by a grouping, with about 1%, 6%, and 14% marking small, medium, and large. Correlation coefficients run from -1 to +1, and Cramér's V from 0 to 1. Odds ratios and rate ratios measure the size of effects in regression. Always report the effect size next to the significance level, and interpret it against what would matter in practice.
Before choosing any test, classify your outcome variable, because the type narrows the options more than anything else.
Numeric variables, such as income, age, or a test score, have meaningful distances between values and can be averaged. Binary variables have two categories, such as yes and no. Nominal variables have several categories without an order, such as region or occupation. Ordinal variables have ordered categories where the gaps are unknown, such as a satisfaction rating or an education level. Counts are whole numbers of events, such as clinic visits or incidents, and are bunched near zero with a long tail.
A single Likert item is ordinal. A score built by summing several items behaves closely enough like a numeric variable that it is usually analysed as one. Coded categories such as region 1 to 5 are never numeric, however they are stored.
Each test rests on assumptions. Parametric tests, such as the t-test, ANOVA, and linear regression, assume that the outcome is roughly normally distributed within groups, that groups have similar spread, and that observations are independent. Non-parametric tests, such as Mann-Whitney U and Kruskal-Wallis, replace the values with their ranks and drop the normality assumption. They suit ordinal outcomes, heavily skewed variables, outliers, and small samples where normality cannot be relied on.
Small departures rarely matter, and the central limit theorem helps here: averages from reasonably large samples are close to normal even when the underlying data are not. As a rough guide, with more than about 30 cases per group the parametric tests are forgiving. With ordinal outcomes, or with small samples that are clearly skewed or contain outliers, use the rank-based version. A small sample that looks roughly symmetric is still better served by the parametric test, which has more power. When both are possible, running both is a useful check, because agreement means your conclusion does not depend on the assumption. What are non-parametric tests? explains how rank-based tests work, when to choose them, and which one matches each parametric test.
Independence is the assumption that matters most and is the hardest to repair. Respondents from the same family, measurements on the same person, and students in the same school are not independent. Tests built for independent observations understate the uncertainty when applied to such data.
Three questions narrow the choice quickly. What type of outcome are you studying? How many groups or measurements are involved? And are the observations independent, or repeated measures on the same people? Figure 1 puts these together for tests that compare groups, measurements, or percentages. Each test in the chart links to its own guide.
For a numeric outcome, the design comes first. One group against a known value uses the one-sample t-test. Two separate groups use the independent t-test. The same people measured twice use the paired t-test. Three or more separate groups use one-way ANOVA, or two-way ANOVA when two grouping factors act together. The same people measured three or more times use repeated-measures ANOVA. In each case, if the outcome is ordinal, badly skewed, or the sample is small, switch to the rank-based partner: Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis, or the Friedman test.
For categories, the chi-square test asks whether two categorical variables are related, and the z-test for proportions tests a percentage against a target or compares two percentages.
When the aim is to model a relationship or make a prediction, the outcome type decides the model, as in Figure 2.
A numeric outcome uses linear regression, or hierarchical regression when predictors are entered in planned blocks. A yes or no outcome uses logistic regression, with probit regression as a close alternative. Unordered categories use multinomial regression and ordered categories use ordinal regression. Counts use Poisson regression, or negative binomial regression when the spread is much larger than the average. When respondents are nested in groups, multilevel regression adds the group structure to a numeric, yes or no, or count model. To measure only how strongly two variables move together, use Pearson or Spearman correlation.
At the 5% level, about one test in twenty will look significant by chance alone, so running many tests almost guarantees a false finding. This is the multiple comparisons problem. It is most visible after an ANOVA, where comparing every pair of groups creates many tests at once. A Bonferroni adjustment multiplies each p-value by the number of tests, a strict but simple fix. The better protection is discipline: decide your comparisons before looking at results, and report how many tests you ran.
A raw difference between two groups can be produced by something else that differs between them. If people who attended a course also had higher incomes to begin with, a simple comparison of outcomes mixes the effect of the course with the effect of income. Such a variable is a confounder. Regression addresses this by estimating each predictor's association with the outcome while holding the others constant. The estimate is then said to be adjusted. Adjustment only works for variables you measured and included, so it reduces confounding without removing it.
Sometimes the effect of one variable depends on another. A teaching method might raise scores for one age group and lower them for another. This is an interaction, and it matters because averaging over it can produce a main effect that describes nobody. Two-way ANOVA tests interactions between two factors directly, and regression models can include them explicitly.
Data are clustered when respondents are nested in groups such as schools, workplaces, or clinics, and repeated when the same units are measured more than once. In both cases observations within a group resemble each other, which the intraclass correlation quantifies. Ignoring this makes standard errors too small and results look more significant than they are. The design effect, roughly 1 plus the average cluster size minus 1, multiplied by the intraclass correlation, shows how much the effective sample shrinks. Multilevel models and repeated-measures methods are built for this structure.
Linear regression models an average directly. A yes or no outcome cannot be modelled that way, because predictions would fall below 0% or above 100%. Generalised linear models solve this with a link function that connects the predictors to the outcome on a suitable scale. Logistic regression uses the logit link and reports odds ratios. Probit uses the cumulative normal. Poisson and negative binomial use a log link and report rate ratios. Ordinal and multinomial models extend the logic to several categories. Knowing the scale of a coefficient, a difference in units, an odds ratio, or a rate ratio, is what allows it to be described correctly.
Measures of fit describe how well a model matches the data. R-squared gives the share of variation explained in linear regression, and pseudo R-squared serves a similar purpose for other models, though its values are not comparable across model types. AIC balances fit against complexity, and among models fitted to the same data, a lower value is preferred. Adding predictors always improves fit in the sample, and a model with too many predictors for its sample memorises noise, which is called overfitting. A common guide is at least 10 to 15 observations per predictor in linear models, and at least 10 events per predictor in logistic models.
Most tests drop any case with a missing value on a variable in the analysis. This shrinks the sample and can bias results if the people with missing values differ from the rest. Check how many cases remain, look at whether missingness is related to the outcome, and report it.
Standard tests assume simple random sampling. Surveys that use stratification, clustering, or unequal selection probabilities need weights and design-aware standard errors, or the uncertainty will be misjudged. If your data comes from such a design, say whether the analysis accounts for it, and treat results from unweighted tests as approximate.
An inferential test can show that two things move together. Only the study design can support a claim that one causes the other. Random assignment is the strongest design. Comparison groups, before and after measures, and careful adjustment improve credibility, but they do not close the gap entirely.
A complete report states the question, the test, the sample size, the estimate with its confidence interval, the effect size, and the significance level, in plain words. State any assumption checks, any adjustments for multiple tests, and any limits on generalising. Give percentages and averages so readers can see the pattern, and reserve statistics such as t and F for technical appendices.
Treating a non-significant result as proof of no difference. It means the data did not provide enough evidence, which is a weaker statement, particularly in small samples.
Ignoring the design. Measuring the same people twice and then using a test for independent groups discards information and can mislead. Match the test to how the data was collected.
Testing without a plan. Trying many comparisons and reporting the significant ones produces false findings.
Reporting significance without size. A tiny effect in a huge sample is not important, and a large effect in a small sample is not nothing.
Reading association as causation. Only the design supports a causal claim.