What is logistic regression? Theory, calculation and application

01What is logistic regression?

Logistic regression is a method for studying yes or no outcomes. It shows which factors are linked to the outcome, how strongly, and how likely the outcome is for a particular person.

Many outcomes have only two possible results: a student passed an exam or did not, a patient recovered or did not, a customer renewed a subscription or did not. The thing you want to explain is called the outcome, and the factors you use to explain it are the predictors. The result you are interested in, such as "passed", is the event.

An example

Suppose you follow 400 students. For each one you know whether they passed an exam, how many hours a week they studied, their attendance rate, and their gender. About 23% of them passed. You want to know three things. Does more study time raise the chance of passing? Do women and men differ once study time and attendance are taken into account? And what is the chance of passing for a woman who studies 10 hours a week with 82% attendance? Logistic regression answers all three.

Why not use an ordinary straight line?

Linear regression fits a straight line to a numeric outcome. For a yes or no outcome, coded 1 for yes and 0 for no, a straight line has two problems. It can predict values below 0% or above 100%, which are impossible chances. And the real pattern is not straight. The chance rises slowly at first, quickly in the middle, and flattens as it approaches certainty. A curve shaped like a stretched letter S fits that pattern, and logistic regression fits such a curve.

What you get out of it

1. Direction
   Whether each predictor raises or lowers the chance of the event.
2. Size
   How big the effect is, usually as an odds ratio, which a later section explains.
3. Confidence
   Whether the pattern could easily be chance, shown by a significance level and a confidence interval.
4. Prediction
   The chance of the event for any combination of predictor values.

02When to use it

Use logistic regression when the outcome has exactly two categories and you want to know which factors are associated with it, how large those associations are, or which characteristics predict the outcome. Predictors can be numeric, categorical, or both.

For a simple relationship between two categorical variables, the chi-square test is enough. For three or more unordered outcome categories, use multinomial regression. For ordered categories, use ordinal regression. Probit regression is a close alternative that gives very similar results.

03Chance, odds and log-odds

Logistic regression describes how likely something is in three connected ways: as a chance, as odds, and as log-odds. Seeing how they link makes the rest of the guide much easier.

Chance

A chance, or probability, is a number from 0 to 1, or from 0% to 100%. If 75 of 100 students passed, the chance of passing is 0.75, or 75%.

Odds

Odds compare the chance that the event happens with the chance that it does not. For a 75% chance, that is 0.75 divided by 0.25, which is 3. People say "3 to 1", meaning three passed for every one who did not. Odds of 1 mean an even chance, odds below 1 mean the event is less likely than not, and odds above 1 mean it is more likely.

Log-odds

The log-odds are the natural logarithm of the odds. They sound abstract, but they have one useful property. They can take any value from very negative to very positive, with zero meaning an even chance, so a straight line can model them without ever producing an impossible result. Converting back to a chance at the end keeps the final answer between 0% and 100%.

Notice the symmetry. A 25% chance and a 75% chance sit the same distance either side of zero.

Putting the three together

Logistic regression models the log-odds as a straight-line combination of the predictors, then converts the answer back into a chance. That two-step route is what keeps every prediction between 0% and 100% and gives the S-shaped curve. The next section writes this as an equation and shows how the numbers are calculated.

04The model and how it is calculated

From a straight line to a probability

The two steps described above become two equations. The first models the log-odds as a straight-line combination of the predictors. The second turns the log-odds back into a probability.

Raising e to the power of a coefficient gives its odds ratio. The probability curve is S-shaped, flattening towards 0% and 100%, so a one-unit change in a predictor moves the probability by different amounts depending on where a person starts.

How the coefficients are found

There is no direct formula, as there is in linear regression. Maximum likelihood is used instead. The software tries sets of coefficients and keeps the set that makes the observed outcomes most probable. The measure of how probable is the log-likelihood, and the search repeats until it stops improving. If a predictor perfectly separates the outcomes, the search never settles, which is the separation problem the module warns about.

Making a prediction

Take a student who studies 10 hours a week, has 82% attendance, and is a woman, using the coefficients from the results below.

1. Log-odds
   -8.367 + 0.137 × 10 + 0.064 × 82 + 0.669 × 1 = -1.042.
2. Odds
   e^-1.042 = 0.353, which is about 1 pass for every 2.8 fails.
3. Probability
   0.353 ÷ (1 + 0.353) = 0.261, or 26%.

For a man with the same hours and attendance, the log-odds are -1.711, the odds 0.181, and the probability 15%. The ratio of the two odds is 0.353 ÷ 0.181 = 1.95, which is the odds ratio for gender in the results. The ratio of the probabilities is only 1.70, which shows why an odds ratio should not be read as "times more likely".

Testing the model

In the example, the model has a log-likelihood of -193.34 and the no-predictor model -214.50. So LR chi-square = 2 × (-193.34 - (-214.50)) = 42.30, compared with a chi-square curve with 3 degrees of freedom, and pseudo R² = 1 - -193.34 ÷ -214.50 = 0.099.

05Reading the output

Odds ratios in plain language

An odds ratio compares the odds between two situations.

An odds ratio of 1 means no difference. Above 1 means higher odds of the outcome, and below 1 means lower odds. If the odds ratio for girls compared with boys is 1.5, the odds of passing are 50% higher for girls. An odds ratio of 0.6 means 40% lower odds. For a numeric predictor, the odds ratio applies to each one-unit increase, such as one additional year of parental education.

Odds ratios are not the same as risk ratios. When the outcome is rare, they are close. When the outcome is common, the odds ratio overstates the change in the chance itself. If the pass rate is already 80%, an odds ratio of 2 does not mean the chance doubles, since it cannot exceed 100%. AnalyZ Solutions phrases large ratios as times more or less likely and smaller ones as percentage differences, and the description of the effect should be read with this in mind.

Coefficients and odds ratios

The coefficient table shows each predictor's odds ratio with a 95% confidence interval and significance level. A toggle switches the table between odds ratios and raw coefficients, which are on the log-odds scale. Odds ratios are easier to explain than raw coefficients. An interval that includes 1 corresponds to a result that is not significant at the 5% level.

Model fit

The summary reports the number of cases, the log-likelihood, a likelihood-ratio test of whether the predictors together improve on a model without them, a pseudo R-squared, AIC, and the percentage of cases classified correctly. Pseudo R-squared is not comparable to R-squared in linear regression. Values of 0.2 to 0.4 are often considered a good fit. Classification accuracy needs care: if 85% of respondents passed, a model that predicts everyone passed is already 85% accurate.

Warnings

If the model has trouble converging, or coefficients are extremely large, the module warns of possible separation or severe overlap between predictors. Separation happens when a predictor perfectly predicts the outcome, for example when nobody in one category experienced the event. The coefficients for such predictors cannot be estimated reliably.

06How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Regression, then Logistic Regression.

1. Select the dependent variable
   The list shows only variables with exactly two categories.
2. Choose the event of interest
   Pick which value counts as the outcome you are modelling, such as Passed or Yes. Odds ratios describe the odds of this event.
3. Select the independent variables
   Choose the predictors, numeric or categorical.
4. Set reference categories
   Categorical predictors are converted to comparisons against a reference category automatically. Pick the reference category you want each group compared with, usually the largest or the most natural comparison group.
5. Apply a subsample filter if needed.
6. Click Run regression
   You get the coefficient table with an odds ratio and coefficient toggle, a model summary, a forest plot of the odds ratios, and a plain-language interpretation.

07Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 400 students are followed to see whether they passed an exam. The event of interest is Passed, and the predictors are weekly study hours, attendance rate, and gender (reference category: Male).

Reading it step by step

1. Check the event and the fit
   The odds ratios describe the odds of passing, because Passed was the chosen event. The likelihood-ratio test has a p-value below 0.001, so the predictors improve on a model with none. Pseudo R² is 0.099, a modest fit.
2. Read a numeric predictor
   The odds ratio for weekly study hours is 1.147. Each extra hour of study goes with about 15% higher odds of passing, holding the other predictors constant. The interval, 1.070 to 1.229, is entirely above 1, and the result is significant at the 0.1% level. Attendance works the same way: each extra point of attendance goes with about 7% higher odds.
3. Read a categorical predictor
   The odds ratio for women compared with men is 1.952, with an interval of 1.178 to 3.235 and a p-value of 0.0094. Women have about 95% higher odds of passing than men with the same study hours and attendance. The interval is wide, so the true gap could be small or large. This is a ratio of odds. It does not mean women are 95% more likely to pass.
4. Treat accuracy with care
   The model classifies 76.25% of students correctly, but 77.25% of students failed. A model that predicted failure for everyone would score slightly higher. Accuracy says little here, and the odds ratios and the likelihood-ratio test are the better guide.

In plain words

Study time, attendance, and gender were each associated with passing the exam. Every extra weekly hour of study raised the odds of passing by about 15%, each attendance point by about 7%, and women had about 95% higher odds than men, all significant after adjustment. The model gives a modest fit, and it describes associations in this sample without proving causes.

08Common mistakes

Describing odds ratios as probabilities. An odds ratio of 2 does not mean twice as likely, especially for common outcomes.

Too few events. A useful guideline is at least 10 events, and 10 non-events, per predictor. Rare outcomes need small models.

Ignoring separation warnings. Merge sparse categories or drop the offending predictor.

Reading accuracy alone. Compare it with the share of the most common outcome.

Treating association as causation. The model describes patterns in observed data.