How to calculate inter-rater reliability in qualitative research

Inter-rater reliability (IRR) is a measure of how consistently two independent researchers apply the same coding scheme to the same data. It is one of the most important quality checks in qualitative analysis, particularly for studies that will be peer-reviewed, published, or used to inform policy decisions.

01Why inter-rater reliability matters

Qualitative coding involves interpretation. Two trained researchers reading the same passage may legitimately disagree about which code applies, particularly for codes with fuzzy boundaries or overlapping definitions. That disagreement is not necessarily a problem: it is often the starting point for a productive conversation about what the code really means.

The problem arises when disagreement is systematic and unexamined. If Coder A applies "Perceived barrier" to any mention of difficulty, while Coder B applies it only to explicit statements of inability, their coded datasets are not comparable. Any frequency analysis, comparative analysis, or thematic claim built on both datasets will be unreliable.

Measuring IRR forces the issue. It tells you which codes are being applied consistently and which are not, so you can resolve disagreements before analysis proceeds.

02What Cohen's Kappa measures

Cohen's Kappa (κ) measures agreement between two coders, corrected for the agreement that would occur by chance alone. If two coders randomly assigned codes, some passages would receive the same code by accident. Kappa subtracts this chance agreement from the observed agreement, giving a more conservative measure of true consistency.

The formula is: κ = (observed agreement − chance agreement) / (1 − chance agreement)

A Kappa of 1.0 means perfect agreement. A Kappa of 0 means the coders agree only as often as chance would predict. A negative Kappa means agreement is worse than chance, which almost always indicates a misunderstanding of the coding task.

The widely accepted minimum for qualitative research reporting is κ ≥ 0.60. Studies published in peer-reviewed journals typically report κ ≥ 0.70 or above.

03How to calculate Kappa in AnalyZ Solutions

1. Each coder completes their independent codingBoth coders code the same set of transcripts independently, without discussing their decisions. This independence is essential: if coders consult each other during coding, the reliability calculation is meaningless.
2. Each coder exports their coding as JSONIn Thematic Coding, go to Review Codes and click the JSON (IRR) export button. This downloads a structured file containing all coded segments, the codes applied to each, and the codebook. Both coders export their own file separately.
3. Open Compare Coding from qualitative moduleCompare Coding is a standalone module accessible directly from the qualitative module cards. It does not require transcripts to be loaded. Make sure both coders' files are available in the same computer.

4. Load both export filesUpload each coder's JSON file and give each a label (e.g. the coder's name or initials). The labels appear in the results table.
5. Click Compute agreement scoreAnalyZ Solutions compares the two files segment by segment, calculates Cohen's Kappa for each code individually, and computes a macro-average Kappa across all codes.

04Reading the results

The results screen shows:

• Overall Kappa (macro-average): the average Kappa across all codes. This is the headline figure, but individual code Kappas are usually more informative.
• Per-code Kappa: a bar showing each code's Kappa and its strength label. This immediately identifies which codes are driving any disagreement.
• Percentage agreement: the raw proportion of segments where both coders agreed, shown alongside Kappa. Useful for context: a code with only 3 segments may show 100% agreement but the Kappa is not meaningful at that sample size.

The interpretation panel below the results classifies each code as strong (κ ≥ 0.80), acceptable (0.60–0.79), or below threshold (κ < 0.60), and provides specific next steps for each category.

05What to do when agreement is low

Low Kappa for a specific code almost always traces back to one of three causes: the code definition is ambiguous, the coders understood the definition differently, or the code is genuinely difficult to apply consistently to this type of data.

1. Compare the two coders' segments side by side
   Export both files and look at which passages one coder coded that the other did not. Look for a pattern: is one coder coding more broadly? More narrowly? Using the code for a different type of passage?
2. Discuss the disagreements
   Bring the disagreeing passages to a calibration session. For each passage, each coder explains why they made the decision they did. These conversations almost always surface the implicit difference in understanding.
3. Revise the code definition
   Update the code description and analytical memo to reflect the agreed definition. Add concrete examples of what qualifies and what does not. Make the boundary cases explicit.
4. Recode a subset of transcripts
   After calibration, have both coders independently recode two or three transcripts using the revised definition. Re-export and re-compute Kappa to confirm improvement before recoding the full dataset.

06When to use IRR and when it is not required

IRR is most important in studies where:

• Two or more researchers are contributing to the coding and analysis
• The findings will be peer-reviewed or published
• The coding framework will be applied to additional data in the future
• The study makes strong causal or prevalence claims based on coded data

Single-researcher studies, which are common in applied evaluation, can report rigour through other means: transparent codebook documentation, analytical memos, member checking, and reflexivity statements. IRR is a tool for demonstrating consistency between coders, not the only marker of rigour in qualitative research.