Volume 1 — Analysis of Categorical Ratings
Volume 1 covers chance-corrected agreement coefficients for categorical data — Cohen's Kappa, Gwet's AC1/AC2, Krippendorff's alpha, and related measures — along with the experimental design, statistical inference, and benchmarking needed to use them correctly.
This edition splits the previous single-volume Handbook into two focused books: Volume 1 for categorical ratings, Volume 2 for quantitative ratings and intraclass correlation. New to this edition is a full chapter on structuring a ratings database before analysis, an expanded treatment of sample-size and power calculations, and new material on inter-annotator agreement for NLP and text-analytics applications.
Nine chapters and three appendices, free
Full PDF excerpts covering the book's foundational and applied chapters, plus reference appendices, exactly as they appear in the book.
Introduction
What inter-rater reliability means in practice, how to scope and design a reliability experiment, and how to choose the right agreement coefficient for your data.
Download PDF ↓Setting Up a Database of Ratings for Analysis
New to the 5th edition. Practical guidance on organizing rating data — contingency tables, wide and long formats — before any coefficient is computed.
Download PDF ↓Agreement Coefficients for Nominal Ratings: A Review
A critical review of Cohen's Kappa, Scott's Pi, Krippendorff's alpha, Gwet's AC1, and the G-Index — how each defines chance agreement, and where each one breaks down.
Download PDF ↓Agreement Coefficients for Ordinal, Interval and Ratio Data
Extends Chapter 3's coefficients beyond nominal categories using weights and Euclidean distance, so partial agreement on ordered scales is credited properly.
Download PDF ↓Constructing Agreement Coefficients: AC1 and Aickin's α
The theoretical foundation behind Gwet's AC1 and Aickin's alpha — what problem each was built to solve, and why they hold up under conditions where Kappa doesn't.
Download PDF ↓Agreement Coefficients and Statistical Inference
How to treat agreement coefficients as estimates with standard errors, and how to calculate the sample size a reliability study actually needs.
Download PDF ↓Benchmarking Inter-Rater Reliability Coefficients
How to interpret a coefficient's magnitude as poor, good, or excellent — a review of existing benchmark scales and a more efficient, coefficient-specific alternative.
Download PDF ↓Inter-Rater Reliability: Conditional Analysis
Agreement conditioned on specific categories, plus validity coefficients for measuring agreement against a gold standard when one exists.
Download PDF ↓Analysis of Nominal-Scale Inter-Rater Reliability Data
Inter-annotator agreement for NLP, testing two coefficients for statistical significance, influence analysis, intra-rater reliability, and Cronbach's alpha.
Download PDF ↓Data Tables
Reference datasets used throughout the book's worked examples, collected in one place for readers who want to reproduce the calculations themselves.
Download PDF ↓Software Solutions
A survey of software options for computing inter-rater reliability coefficients — R, AgreeStat for Excel, online calculators, SAS, and SPSS/STATA.
Download PDF ↓Sample Size Calculations
Supporting datasets and worked examples for the power and sample-size methods introduced in Chapter 6.
Download PDF ↓Full table of contents
- 1 Introduction2
- 2 Setting Up a Database of Ratings for Analysis37
- 3 Agreement Coefficients for Nominal Ratings: A Review54
- 4 Agreement Coefficients for Ordinal, Interval and Ratio Data100
- 5 Constructing Agreement Coefficients: AC1 and Aickin's α138
- 6 Agreement Coefficients and Statistical Inference169
- 7 Benchmarking Inter-Rater Reliability Coefficients219
- 8 Inter-Rater Reliability: Conditional Analysis240
- 9 Analysis of Nominal-Scale Inter-Rater Reliability Data284
- A Data Tables340
- B Software Solutions349
- C Sample Size Calculations367
- Bibliography376
- List of Notations385
- Author Index389
- Subject Index393
Looking for quantitative ratings instead?
Volume 2 covers Intraclass Correlation Coefficients (ICC) for interval and ratio data — the companion to this book for researchers working with continuous measurements rather than categories.
See Volume 2