How to measure annotator agreement: Cohen’s kappa and Krippendorff’s alpha

Measure annotator agreement with a chance-corrected statistic: Cohen’s kappa (κ) when two annotators label the same items, and Krippendorff’s alpha (α) when you have more annotators, missing labels, or ordinal and numeric scales. Raw percent agreement overstates reliability, because annotators agree some of the time by chance.

Agreement is the main evidence that a label means something. If two qualified experts can’t agree which of two answers is better, a reward model trained on their labels learns noise. Here’s how to compute both statistics, with worked examples you can check by hand.

Why isn’t percent agreement enough?

Two annotators labelling at random still agree some of the time — half the time on a balanced yes/no task — and percent agreement can’t tell skill from luck. It gets worse when one label dominates. Suppose two reviewers screen 100 model responses for safety:

• Both say “safe”: 90
• Both say “unsafe”: 2
• They disagree: 8 (4 each way)

They agree 92% of the time, which sounds excellent. But each reviewer calls 94 of the 100 responses safe, so they’d agree about 88.7% of the time by chance alone (0.94² + 0.06²). On the cases that matter — the unsafe ones — they rarely agree. Cohen’s kappa for this table is 0.29.

How do you calculate Cohen’s kappa?

Kappa compares observed agreement with the agreement expected by chance, given each annotator’s own label frequencies:

κ = (p_o − p_e) / (1 − p_e)

Here p_o is the observed share of agreement and p_e the share expected by chance. A worked example: two reviewers, Maya and Daniel, judge the same 50 preference pairs, each deciding whether response A or response B is better.

1. Observed agreement. They agree on 20 + 15 = 35 of 50 pairs, so p_o = 0.70.
2. Chance agreement. Maya picks A 60% of the time and Daniel 50%. The chance both pick A is 0.6 × 0.5 = 0.30; both pick B, 0.4 × 0.5 = 0.20. So p_e = 0.30 + 0.20 = 0.50.
3. Kappa. κ = (0.70 − 0.50) / (1 − 0.50) = 0.20 / 0.50 = 0.40.

Seventy percent agreement becomes a kappa of 0.40: Maya and Daniel achieve 40% of the agreement that was possible beyond chance. On the widely quoted Landis and Koch scale that’s the top of the “fair” band (0.21–0.40), just short of “moderate”. Treat those bands as rough conventions, not standards.

When should you use Krippendorff’s alpha instead?

Kappa is built for exactly two annotators who label every item with categories. Krippendorff’s alpha generalizes in three directions:

• Any number of annotators, including different people on different items.
• Missing labels: an item labelled by only some annotators still counts, as long as at least two labelled it.
• Any level of measurement: nominal categories, ordinal scales such as 1–5 rubric scores, or interval values, each with its own distance function.

It’s defined as α = 1 − D_o / D_e: one minus the ratio of the disagreement observed to the disagreement expected by chance.

How do you calculate Krippendorff’s alpha? A worked example

Three graders — Maya, Daniel and Elena — mark six agent responses Pass or Fail against a rubric. Elena didn’t grade response 4.

Step 1: build the coincidence matrix. For each response, count every ordered pair of labels from different graders, weighted by 1/(m − 1), where m is the number of labels that response received. Response 1 has three Pass labels, which make six ordered Pass–Pass pairs at weight ½: 3 Pass–Pass. Response 2 (Pass, Pass, Fail) contributes 1 Pass–Pass, 1 Pass–Fail and 1 Fail–Pass. Response 4 has only two labels, so its pairs get weight 1: 1 Pass–Fail and 1 Fail–Pass. Summing all six responses:

The row totals are simply the number of Pass labels (10) and Fail labels (7) among the 17 labels given.

Step 2: observed disagreement. D_o = (3 + 3) / 17 = 0.353.

Step 3: expected disagreement. If the 17 labels were paired at random, D_e = (10 × 7 + 7 × 10) / (17 × 16) = 140 / 272 = 0.515.

Step 4: alpha. α = 1 − 0.353 / 0.515 = 1 − 0.686 = 0.314.

The graders’ weighted pairwise agreement is (7 + 4) / 17 = 65%, yet alpha is only 0.31. Krippendorff recommends α ≥ 0.800 for data you rely on and treats 0.667 as the lowest level for tentative conclusions. This rubric needs work before anyone trains on it.

For ordinal or numeric scales, replace the 0-or-1 disagreement with a distance between scale points, so a 4-versus-5 split costs less than a 1-versus-5 split. And as a sanity check, alpha on Maya and Daniel’s two-reviewer table also comes out at 0.40: with two annotators and complete data, alpha and kappa usually land close together.

How is agreement different from gold accuracy?

Agreement measures consistency; gold accuracy measures correctness. Two annotators can agree perfectly and both be wrong — for instance, when the guideline itself is wrong. Use both:

• Gold tasks — items with known answers, mixed invisibly into the queue — catch individuals who are wrong.
• Agreement catches tasks and guidelines that are ambiguous.

If agreement is low but gold accuracy is high, your gold items are probably too easy. If both are low, fix the guidelines before buying more labels.

How should you report agreement?

• Per project and per criterion. A rubric with one ambiguous criterion can hide behind a healthy average.
• With the sample described. Say how many items were double-labelled and how they were chosen — at random, not the easy ones.
• With uncertainty. Bootstrap a confidence interval by resampling items; on 50 items, kappa’s interval is wide.
• Next to raw agreement and label frequencies, so readers can spot the prevalence effect from the safety example.
• Over time. Drift usually means the guidelines or the mix of items changed.

For context, OpenAI’s GDPval reported 71% agreement between human graders. In Sandworld’s data projects, double-reviewed items feed per-project κ and α alongside each expert’s gold accuracy, so reliability is visible next to volume. For the buying side, see how to buy expert training data.