What is linear regression? Theory, calculation and application

01What is linear regression?

Linear regression is a method for studying how one numeric outcome changes with other factors. It shows which factors are linked to the outcome, by how much, and what value to expect for a particular person.

The thing you want to explain, such as income, a test score or monthly spending, is called the outcome. The factors you use to explain it, such as years of education or age, are the predictors. A coefficient is the number that tells you how much the outcome changes when a predictor goes up by one unit.

An example

Suppose you have information on 300 adults: their monthly income, years of education, age and gender. You want to know three things. How much more do people earn for each extra year of education? Do women and men earn different amounts once education and age are taken into account? And what income would you expect for a woman aged 35 with 12 years of education? Linear regression answers all three.

Why not use a correlation or a comparison of averages instead?

A correlation says whether two variables move together, but it gives no amount and cannot account for anything else. Comparing the average income of women and men ignores that the two groups may differ in education and age. Regression handles both problems. It states the effect of each factor in the outcome's own units, and it separates the factors by comparing people who are alike on all the others.

What you get out of it

1. Direction
   Whether each predictor goes with a higher or lower outcome.
2. Size
   How much the outcome changes for each unit of the predictor, in the outcome's own units.
3. Confidence
   Whether the pattern could easily be chance, shown by a significance level and a confidence interval.
4. Fit
   How much of the variation in the outcome the predictors explain together, called R-squared.
5. Prediction
   The expected outcome for any combination of predictor values.

02When to use it

Use linear regression when the outcome is numeric and roughly continuous, such as income, a test score, a wellbeing index, or monthly spending. Predictors can be numeric, categorical, or a mix. It suits questions like which factors are associated with higher scores, how large the gap between groups is after accounting for other differences, and how much of the variation the predictors explain together.

For yes or no outcomes, use logistic regression. For counts, use Poisson or negative binomial regression. For ordered categories, use ordinal regression. For people nested in groups, use multilevel regression.

03The building blocks: a line, a slope and an error

Linear regression rests on three ideas: a straight line, the two numbers that describe it, and the gap between the line and real data. To keep the arithmetic visible, this section and the worked calculation below use six people. The results shown later use a larger set of 300.

The straight line

A straight line is described by two numbers. The intercept is where the line starts, meaning the predicted outcome when the predictor is zero. The slope is how steeply it rises, meaning the change in the outcome for each one-unit increase in the predictor. For the six people, the line that fits best is:

A person with 12 years of education is predicted to earn 680 + 160 × 12 = 2,600. Each extra year adds 160. The intercept of 680 describes someone with no education at all, which lies outside the range of the data. Treat it as the starting point of the line and not as a real prediction.

The error

Real people do not fall exactly on the line. The gap between what a person actually earns and what the line predicts is the residual. The part of the outcome that the predictors do not explain is called the error. Regression looks for the line that makes these gaps as small as possible overall.

More than one predictor

A model can use several predictors at once, with one slope for each. A slope is also called a coefficient.

Each slope is the effect of one predictor among people who are the same on the others. A category such as gender is entered as a comparison with a chosen reference group, so its slope is the difference between the groups. The next section writes the model in symbols and shows how the best line is found.

04The model and how it is calculated

The model

The building blocks above combine into one equation: the outcome is a baseline plus a contribution from each predictor, plus whatever the predictors fail to explain.

The intercept b0 is the predicted outcome when every predictor is zero. Each coefficient b is the change in the predicted outcome for a one-unit increase in that predictor, holding the others constant. The error is the part of each person's outcome that the predictors cannot account for.

How the coefficients are found

The method of least squares chooses the coefficients that make the sum of the squared residuals as small as possible, where a residual is the gap between a person's observed outcome and the value the line predicts. Squaring stops positive and negative gaps from cancelling out and gives large misses extra weight. With one predictor there are direct formulas:

With several predictors the principle is the same, and software solves for all the coefficients at once using matrix algebra.

A worked example with one predictor

1. Averages
   x-bar = 12 years and y-bar = 2,600.
2. Slope
   The products add up to 6,400 and Σ(x - x-bar)² = 40, so b1 = 6,400 ÷ 40 = 160. Each extra year of education goes with 160 more units of monthly income.
3. Intercept
   b0 = 2,600 - 160 × 12 = 680.
4. Predictions and residuals
   Someone with 12 years is predicted to earn 680 + 160 × 12 = 2,600. The residuals in the last column add up to zero, which is always true of a least squares line with an intercept.
5. How well it fits
   The squared residuals add up to 216,000, against a total variation of 1,240,000 around the average. R² = 1 - 216,000 ÷ 1,240,000 = 0.826, so the line explains about 83% of the variation.
6. Is the slope real
   The standard error of the slope is √(216,000 ÷ 4 ÷ 40) = 36.74, so t = 160 ÷ 36.74 = 4.35 with 4 degrees of freedom, and p = 0.012.

R², adjusted R², and the F test

Here n is the number of cases and k the number of predictors. Adjusted R² penalises predictors that add little. The F test asks whether the predictors together explain more than chance would. For the three-predictor income model in the results below, with n = 300 and R² = 0.245, F = (0.245 ÷ 3) ÷ (0.755 ÷ 296) = 32.08, which matches the table.

Making a prediction

Once the coefficients are known, a prediction is just arithmetic. With the model in the results below, a woman with 12 years of education aged 35 is predicted to earn:

Predictions are only trustworthy within the range of the data used to fit the model.

05Reading the output

Coefficients

Each predictor has a coefficient, labelled B. For a numeric predictor, B is the expected change in the outcome for a one-unit increase in that predictor, holding the others constant. If B for years of schooling is 42, each extra year goes with 42 more units of monthly income, on average, among people who are otherwise similar.

For a categorical predictor, B compares a category with the reference category. If the reference is Male and B for Female is -35, women earn 35 units less than men on average, after accounting for the other predictors.

The table also gives a standard error, a test statistic, a significance level, and a 95% confidence interval for each coefficient. An interval that includes zero corresponds to a coefficient that is not significant at the 5% level.

R-squared

R-squared is the share of the variation in the outcome that the model explains. AnalyZ Solutions reads it as weak below 10%, modest from 10% to 25%, moderate from 25% to 50%, and strong above 50%. Social research often produces modest values, since human outcomes are shaped by many things a survey does not capture. Adjusted R-squared penalises the addition of predictors that contribute little, and is the better figure when comparing models with different numbers of predictors.

The overall test

The F test asks whether the predictors together explain more than chance would. It is reported with its significance level. A significant model can still have a low R-squared, which just means real but modest predictive power.

06Assumptions

A roughly linear relationship between each numeric predictor and the outcome. Plot them to check.

Independent observations. People sampled in clusters, or measured repeatedly, violate this. Use multilevel regression for such data.

Roughly constant spread of the outcome across the range of predictions.

Approximately normal residuals, which matters mainly in small samples.

Predictors that are not almost duplicates of each other. When two predictors are extremely highly correlated, the model cannot separate their effects, and coefficients become unstable. Check the correlation matrix before including both.

A useful rule of thumb is to have at least 10 to 15 observations for each predictor in the model.

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Regression, then Linear Regression.

1. Select the dependent variable
   Choose the numeric outcome.
2. Select the independent variables
   Choose one or more predictors. Numeric and categorical variables can be mixed.
3. Set reference categories
   Categorical predictors are converted to comparisons against a reference category automatically. Pick the reference category you want each group compared with, usually the largest or the most natural comparison group.
4. Apply a subsample filter if needed
   For example, to fit the model for one region.
5. Click Run regression
   The output includes the coefficient table, a model summary with the sample size, R-squared, adjusted R-squared, the F test and AIC, a coefficient plot, and a plain-language interpretation that leads with model fit and then describes the strongest significant predictor.

You can export the tables as CSV, Excel, or Word, send the results to Sense Making, or add the coefficient plot to a Data Canvas slide.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose monthly income is modelled from years of education, age, and gender (reference category: Male) for 300 people.

Reading it step by step

1. Start with the model fit
   R² is 0.245, so the three predictors together explain about 25% of the variation in income, a modest share. The F test has a p-value below 0.001, so the predictors explain more than chance would. The remaining 75% of the variation reflects things not in the model.
2. Read a numeric predictor
   The coefficient for years of education is 42.99. Each extra year of education goes with about 43 more units of monthly income, holding age and gender constant. The interval, 33.93 to 52.05, is fairly narrow, so the estimate is precise.
3. Read the weaker predictor
   Age has a coefficient of 3.63 with a p-value of 0.0049, so each extra year goes with about 3.6 more units of income. It is significant at the 1% level, and it is small next to education.
4. Read a categorical predictor
   The gender coefficient compares women with men. Women's income is 30.52 lower on average, but the p-value is 0.2127 and the interval, -78.60 to 17.57, includes zero. The data give no clear evidence of a gender gap after adjusting for education and age, though the wide interval means a real gap cannot be ruled out.

In plain words

Education was the strongest predictor of income. Each extra year was associated with about 43 more units, controlling for age and gender, and the model explained about 25% of the variation. Age had a small positive association. Gender showed no clear difference once the other variables were accounted for. These are associations after adjustment, and they do not show that extra education causes higher income.

09Common mistakes

Reading coefficients as causal effects. Regression describes association after adjustment. Causal claims need a design that supports them.

Comparing coefficient sizes across differently scaled predictors. A coefficient for income in thousands cannot be compared directly with one for age in years.

Ignoring the reference category. Every categorical coefficient is a comparison with the reference group, so choose it deliberately and state it.

Overfitting with too many predictors. Small samples with many predictors give unreliable coefficients.

Extrapolating. The model describes the range of your data. Predictions far outside it are guesses.