How to test the reliability of a questionnaire scale

01What is Cronbach's alpha?

Cronbach's alpha answers one question: do the statements in my scale pull in the same direction? If they do, you can add them into a single score. Alpha is a number, usually between 0 and 1. The closer it is to 1, the more the statements agree with each other.

Surveys often ask several questions about one idea. To measure wellbeing, for example, you might ask people how far they agree with six statements, from 1 (strongly disagree) to 5 (strongly agree). The set of statements is called a scale. Before you add the six answers into one wellbeing score, you want to know that the statements belong together. If one of them is really about something else, the total score becomes muddled. Alpha is the usual way to check.

An example

Suppose 250 adults answer six statements about their wellbeing, each from 1 to 5: I feel cheerful, I feel calm, I have energy for the day, I often feel low, I enjoy my daily activities, and I am satisfied with my living space. The fourth statement is worded the opposite way from the others, so a high rating means worse wellbeing. You want one wellbeing score per person. Alpha tells you whether adding the six answers is reasonable, and which statements to look at if it is not.

What reliability means here

A reliable scale gives consistent results. That has two parts: the items agree with each other, and the scale gives similar scores when the same people are measured again. Alpha speaks only to the first part. It does not show that scores are stable over time, and it does not show that the scale measures the right concept. A set of items can agree closely and still measure the wrong thing.

What you get out of it

1. Alpha
   One value for the whole scale, reported both as raw alpha and as standardised alpha.
2. Mean inter-item correlation
   How closely the statements are related to each other on average.
3. Item-total statistics
   For each item, how strongly it relates to the rest of the scale, and what alpha would be without it.
4. Correlation matrix
   Every pair of items side by side, which helps to find items that do not belong or are worded in reverse.

02When to use it

Use alpha when you have two or more numeric items that are meant to measure the same concept, and you plan to combine them into a total or mean score. The items should point the same way, or you should be ready to reverse-score the ones that do not, and they should use the same kind of response scale.

Alpha is not suited to a list of things that need not go together. A count of household assets, or a checklist of services a family used, is an index, and a family can have a radio without having a bicycle. Items like these have no reason to correlate. Alpha is also the wrong tool when the questions cover several different concepts. Run an exploratory factor analysis first to see how many dimensions there are, then calculate alpha for each one. To check agreement between two people who coded the same material, see inter-rater reliability.

03Why items should agree

Each answer to a statement is partly the person's true level of the concept and partly noise: a misread word, a passing mood, a slip of the pen. When you add several items, the true part builds up and the noise partly cancels. Alpha estimates how much of the variation in the total score comes from the shared, true part.

Variance of the total score

The idea can be seen in the variance. If you add two items that are unrelated, the variance of the total is the sum of the two item variances. If the items move together, the variance of the total is larger than that sum, by twice their covariance. The more the items agree, the bigger the gap between the variance of the total and the sum of the item variances. Alpha is built from that gap. With unrelated items the gap is zero and alpha is 0. When the items agree closely, the gap is large and alpha approaches 1.

04How alpha is calculated

The formula

Here k is the number of items. The total score is the sum of the items for each person. The factor k / (k - 1) adjusts for the number of items. The module reports this as the raw alpha, and also a standardised version that works from the average correlation between items instead of variances:

The two values are close when items have similar variances, as they do for items rated on the same scale.

A worked example

Take the six wellbeing items in the example, after the fourth has been reverse-scored. The six item variances add up to 9.737, and the variance of the total score is 26.824.

The mean correlation between pairs of items is 0.350. The standardised version is 6 × 0.350 / (1 + 5 × 0.350) = 0.764.

Length matters

Alpha depends on the number of items as well as on how closely they agree. Adding items raises alpha even if the average correlation between them stays the same. A long scale of weakly related items can reach 0.80, and a short scale of good items can stay under 0.70.

For this reason, read alpha together with the mean inter-item correlation, which the module reports next to it. Clark and Watson (1995) suggest that a mean inter-item correlation between about 0.15 and 0.50 is a sensible target. Much lower means the items share little. Much higher means they may be asking the same question in different words.

Item-total correlations

The corrected item-total correlation is the correlation between one item and the sum of all the other items. It is called corrected because the item is left out of the total. Otherwise the item would be correlated partly with itself, which inflates the value. The module also shows the uncorrected version, labelled item-total r, which is always higher. Use the corrected one to judge items.

Alpha if item deleted is the alpha you would get for the remaining items if you removed that one. If it is higher than the current alpha, the item is pulling the scale down.

05Reading the output

How high should alpha be?

The module labels alpha with the bands below, taken from George and Mallery (2003). They are rules of thumb. Many textbooks accept 0.70 or more for research that compares groups, and ask for 0.90 or more when scores are used to make decisions about individual people. A value above about 0.95 often means that some items repeat each other and the scale could be shorter.

Item-total statistics

The table lists each item with its mean, standard deviation, corrected item-total r, item-total r and alpha if the item is deleted. Corrected item-total values below 0.30 are shown in red, and the same cut-off is drawn on the bar chart of item discrimination, where bars below 0.30 are red. An item below 0.30 does not relate well to the rest of the scale.

The interpretation text

Below the results, the module writes a short summary. It gives a verdict on alpha, says whether the items can be combined into a composite score, lists any items with weak item-total correlation, and names the item whose removal would raise alpha the most. Treat the last point as a prompt to look at the item, not as an instruction to delete it.

Reverse-scored items

If you tick the reverse-coding option, items that were flipped are marked (R) in the table and named above it. A negative correlation between an item and most of the others is the sign that the item is worded in the opposite direction.

Missing data

Only people who answered every selected item are included. The number of cases is shown in the summary, so compare it with your sample size to see how many were dropped.

06How to run it in AnalyZ Solutions

Open Scale & Reliability, then choose Cronbach's Alpha.

1. Select the scale items
   Choose two or more numeric variables that measure the same concept. Include every item of the scale, and only those.
2. Decide on reverse coding
   Tick Adjust for reverse coding if some items are worded in the opposite direction. The module finds items that correlate negatively with most of the others and flips their scores before it calculates alpha.
3. Choose what to show
   Show item-total statistics and Also report standardised alpha are both ticked by default.
4. Apply a subsample filter if needed
   Use it to calculate alpha for one group, such as women only or one district. Reliability can differ between groups.
5. Click Run Cronbach's Alpha
   You get alpha with its label, standardised alpha, the mean inter-item correlation, the item-total table, the discrimination chart, a collapsible correlation matrix and a written interpretation.

You can export the tables as CSV, Excel or Word, or add the result to AnalyZense or to a Data Canvas story.

07Interpreting the result

The figures below are invented to show the steps, and they follow the results tables in AnalyZ Solutions, with a few columns left out to save space. Suppose 250 adults rated the six wellbeing statements from 1 to 5. The fourth statement, I often feel low, is worded in reverse, and the reverse-coding option was ticked.

Reading it step by step

1. Check the reversal
   The module reversed one item, I often feel low, and marked it (R). This matters a great deal. Without the reversal, alpha for the same data is only 0.430, because the reversed item cancels the effect of the others.
2. Read the overall value
   Alpha is 0.764, which the module labels acceptable. The mean inter-item correlation is 0.350, inside the usual range, so the result is not the product of scale length alone.
3. Look at the item-total correlations
   Five items have corrected item-total correlations between 0.52 and 0.64, well above 0.30. The sixth, satisfaction with living space, has 0.25 and is shown in red. Its correlations with the other items in the matrix are only 0.12 to 0.22.
4. Check alpha if item deleted
   For the living space item the value is 0.793, higher than the current 0.764. For every other item it is lower, between 0.692 and 0.725, so removing any of them would weaken the scale.
5. Decide what to do
   Satisfaction with living space is related to wellbeing, but it is about circumstances and not about how a person feels. Dropping it is reasonable, and so is keeping it if the scale is an established one that you want to use unchanged. Whichever you choose, report the alpha with all six items and say what removing the weak one would do.

In plain words

The six wellbeing items form an acceptably reliable scale (alpha = 0.76, N = 250) once the item I often feel low is reverse-scored. One item, satisfaction with living space, is only weakly related to the others (corrected item-total r = 0.25), and removing it would raise alpha to 0.79. Alpha shows that the items agree. It does not show that they measure a single dimension.

08Common mistakes

Forgetting reverse-worded items. One item scored in the wrong direction can drag alpha far below its true value, as the example shows.

Reading a high alpha as proof of one dimension. Items from two related concepts can still give an alpha of 0.80. If you are unsure about the structure, run a factor analysis.

Putting several concepts into one alpha. Calculate it separately for each subscale and report each value.

Chasing a higher number. Removing items until alpha rises, or adding near-copies of existing items, improves the number and damages the scale.

Using alpha for an index. If the items are not expected to move together, a low alpha means nothing.

Trusting a small sample. With fewer than about 50 respondents, alpha is imprecise. Treat it as a rough guide.

Leaving out the details. State the number of items and the number of cases with the value, for example alpha = 0.76, 6 items, N = 250.