What is probit regression? Theory, calculation and application

01What is probit regression?

Probit regression is a method for studying yes or no outcomes. It shows which factors are linked to the outcome, how strongly, and how likely the outcome is for a particular person.

Many outcomes have only two possible results: a student passed an exam or did not, a patient recovered or did not, a person took up an offer or did not. The thing you want to explain is called the outcome, and the factors you use to explain it are the predictors. The result you are interested in, such as "passed", is the event. Probit regression is a close relative of logistic regression, which does the same job through a slightly different curve.

An example

Suppose you follow 400 students. For each one you know whether they passed an exam, how many hours a week they studied, their number of absences, and their gender. You want to know three things. Does more study time raise the chance of passing? Do women and men differ once study time and absences are taken into account? And what is the chance of passing for a woman who studies 10 hours a week and has 4 absences? Probit regression answers all three.

Why not use an ordinary straight line?

Linear regression fits a straight line to a numeric outcome. For a yes or no outcome, coded 1 for yes and 0 for no, a straight line has two problems. It can predict values below 0% or above 100%, which are impossible chances. And the real pattern is not straight. The chance rises slowly at first, quickly in the middle, and flattens as it approaches certainty. A curve shaped like a stretched letter S fits that pattern, and probit regression fits such a curve.

What you get out of it

1. Direction
   Whether each predictor raises or lowers the chance of the event.
2. Size
   How big the effect is, measured as a shift in an underlying tendency, which a later section explains.
3. Confidence
   Whether the pattern could easily be chance, shown by a significance level and a confidence interval.
4. Prediction
   The chance of the event for any combination of predictor values.

02When to use it

Use probit for a binary outcome when your field expects it, or when you have theoretical reasons to think of the outcome as a threshold on an underlying normal tendency. In most applied work, the choice makes little practical difference. The two models classify cases almost identically, and predicted probabilities are nearly the same except in the extreme tails.

The practical difference lies in communication. Logistic regression gives odds ratios that readers can interpret directly. Probit coefficients have no such intuitive translation, which is the main reason logistic regression is often preferred when results must be explained in plain terms.

Probit compared with logistic regression

03The building blocks: chance, the index and the link

Probit regression rests on three ideas: a single number that combines the predictors, a rule that turns it into a chance, and a hidden tendency that explains the rule.

Chance and the index

A chance, or probability, is a number from 0 to 1, or from 0% to 100%. Probit regression combines the predictors into one number called the index, in the same way as an ordinary straight-line model. The index can be any number, large or small, positive or negative, so it cannot be used as a chance directly. Something has to convert it into a value between 0 and 1.

The link

The rule that turns the index into a chance is called the link. Probit regression uses the cumulative normal curve, written Φ, which gives the area under the bell-shaped normal curve up to a given point. It rises smoothly from 0 to 1, so any index becomes a valid chance. The name probit comes from "probability unit". Logistic regression uses a different link, which works through odds, and the two curves have almost the same shape, as a later figure shows.

The latent-variable idea

Imagine that behind the yes or no outcome sits a hidden continuous tendency, such as a student's underlying readiness to pass an exam. In a probit model each person's tendency equals the index plus a random disturbance that follows a standard normal curve. The outcome is yes when the tendency is above zero. The chance of yes is therefore the area of the normal curve that lies above the threshold, which is Φ(index).

The index moves the whole curve. In the figure, the average tendency for a woman is slightly above zero, so 57% of her curve lies above the threshold, while for a man it is slightly below, so 41% does.

The method was developed in the 1930s for dose-response experiments in toxicology, where each organism has a tolerance and responds once the dose passes it. It remains common in economics, where a decision to act is modelled as a tendency crossing a threshold.

The next section writes this as an equation and shows how the coefficients are found.

04The model and how it is calculated

The model

Probit regression builds an index from the predictors and passes it through the cumulative normal curve, written Φ, which turns any number into a probability between 0 and 1.

The latent-variable story behind it is that each person has an unobserved tendency, equal to the index plus a random normal disturbance. The outcome is yes when that tendency passes a threshold. Φ(index) is then the probability that the tendency lands above the threshold.

How the coefficients are found

As in logistic regression, the coefficients come from maximum likelihood. The software searches for the values that make the observed yes and no outcomes most probable, and reports the log-likelihood, a likelihood-ratio test, and a pseudo R².

Making a prediction

Take a student who studies 10 hours a week, has 4 absences, and is a woman, using the coefficients from the results below.

1. Index
   -0.697 + 0.082 × 10 + (-0.087) × 4 + 0.402 × 1 = 0.183.
2. Probability
   Φ(0.183) = 0.573, or 57%.

For a man with the same hours and absences, the index is -0.218 and the probability is 41%. Unlike the logistic case, no single number such as an odds ratio summarises the gap. The change in probability depends on where the person starts, which is why probit results are usually described by direction and significance.

Probit and logistic compared

The two curves have nearly the same shape. Logistic has slightly heavier tails, and the logistic index runs about 1.7 times larger for the same probability. Multiplying the index by 1.7 makes the logistic curve almost match the probit one, which is why logistic coefficients are typically 1.6 to 1.8 times the size of probit ones.

The probit column and the last column agree closely, and the only noticeable gaps are in the far tails.

05Assumptions

Probit regression rests on a few assumptions. Breaking them does not stop the model from running, but it can mislead.

• A binary outcome. Two categories, with the event of interest chosen deliberately.
• Independent observations. Each person appears once and is unrelated to the others. Students in the same school, or patients in the same clinic, need a multilevel model.
• A linear index. The index should change in a straight line with each numeric predictor. A predictor whose effect rises and then falls fits poorly.
• A normal disturbance. The hidden tendency is assumed to be normally distributed around the index. The logistic assumption fits almost as well, which is why the choice rarely changes conclusions.
• No perfect separation. If a predictor splits the outcomes completely, its coefficient cannot be estimated, and the model may fail to settle.
• Enough cases. Aim for at least about 10 cases of the less common outcome for each predictor.

06Reading the output

Coefficients

Probit coefficients are on the scale of the underlying normal tendency, measured in standard deviations of that tendency. In the example, a coefficient of 0.082 for weekly study hours means that each extra hour raises the average tendency to pass by 0.082 standard deviations. A positive coefficient means the predictor raises the chance of the outcome, and a negative one lowers it. The size of a coefficient cannot be turned into an odds ratio or a percentage change directly, because the effect on the chance depends on where the person starts. A given coefficient changes the probability more for someone near the middle of the range than for someone whose chance is already very high or very low.

For reporting, focus on three things: the direction of each coefficient, its significance level and confidence interval, and the relative size of coefficients for predictors on similar scales. AnalyZ Solutions describes significant predictors as raising or lowering the chance of the outcome. The odds ratio toggle is not offered for probit, since odds ratios belong to the logistic model.

Turning coefficients into probabilities

Because the curve is S-shaped, one coefficient moves the probability by different amounts at different starting points. For a small change, the shift in probability is approximately:

Here φ is the height of the normal curve at the person's index. Take a woman who studies 10 hours a week and has 4 absences. Her index is 0.183 and φ is 0.392, so one extra study hour raises her chance of passing by about 0.082 × 0.392 = 0.032, or 3.2 percentage points. For a student whose chance is already high, with an index of 1.17 and φ of 0.20, the same hour adds only about 1.6 points, the exact change shown in the figure.

The curve is steepest where the chance is 50%, because φ is largest there at about 0.4. As a rule of thumb, a coefficient times 0.4 is the largest change in probability that one unit of the predictor can produce. AnalyZ Solutions does not report these changes. Calculate them from the coefficients, or compare predicted probabilities for example profiles, as in the worked prediction above.

Model fit

The summary reports the number of cases, the log-likelihood, a likelihood-ratio test, a pseudo R-squared, AIC, and the percentage of cases classified correctly. These are read as in logistic regression. AIC is useful for comparing probit and logistic fits on the same data, with the lower value indicating the better fit.

07How to run it in AnalyZ Solutions

Open Inferential Analysis, choose Regression, then Probit Regression.

1. Select the dependent variable
   The list shows variables with exactly two categories.
2. Choose the event of interest
   Pick the value that counts as the outcome.
3. Select the independent variables
   Choose numeric or categorical predictors.
4. Set reference categories
   Categorical predictors are converted to comparisons against a reference category automatically. Pick the reference category you want each group compared with, usually the largest or the most natural comparison group.
5. Apply a subsample filter if needed.
6. Click Run regression
   The output includes the coefficient table, model summary, a coefficient plot, and a plain-language interpretation.

08Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 400 students are followed to see whether they passed an exam, using weekly study hours, absences, and gender (reference category: Male) as predictors.

Reading it step by step

1. Check the model as a whole
   The likelihood-ratio test has a p-value below 0.001, so the predictors together improve on a model with none. Pseudo R² of 0.080 is modest. Accuracy is 62.0%, compared with a pass rate of about 50%, so the model does better than guessing.
2. Read the direction of each coefficient
   Study hours have a positive coefficient (0.082) and absences a negative one (-0.087). More study raises the chance of passing, and more absences lower it. Both are significant, at the 0.1% and 1% levels.
3. Read the categorical predictor
   The gender coefficient is 0.402 with a p-value of 0.0020, so women are more likely to pass than men with the same hours and absences. The interval, 0.146 to 0.657, is above zero.
4. Do not translate coefficients into odds ratios
   Probit coefficients sit on the scale of an underlying normal tendency. The size of a coefficient cannot be read as an odds ratio or a percentage change. Use the direction and significance, and compare sizes only between predictors on similar scales. If you need odds ratios, use logistic regression.

In plain words

Study hours and gender raised the chance of passing, while absences lowered it, all significant at the 1% level or better. The direction and significance are the main findings, and the sizes are on a scale that is hard to explain to a general reader. Running logistic regression on the same variables would give similar conclusions with odds ratios that are easier to describe.

09Common mistakes

Interpreting probit coefficients as odds ratios. They are on a different scale.

Reading a coefficient as a change in probability. A coefficient moves the hidden tendency, not the probability. The change in probability depends on where the person starts and is largest near 50%.

Comparing coefficient sizes with logistic coefficients. Probit coefficients are typically about 0.6 times the size of logistic ones for the same data, a difference that comes from the scale of the curve and does not indicate a weaker effect.

Overlooking separation. A predictor that perfectly divides the outcome cannot be estimated, in either model.

Too many predictors for the number of events. The same guidelines as logistic regression apply.