advanced analytics

Handbook of Inter-Rater Reliability (4th Edition)
by Kilem L. Gwet. Ph.D.

Handbook's Main Page Error Page R functions AgreeStat  

Welcome to the website of the 4th edition of the "Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters."

inter-rater reliability

The third edition of this book was very well received by researchers working in many different fields of research. The use of that text also gave these researchers the opportunity to raise questions, and express additional needs for materials on techniques poorly covered in the literature. For example, when designing an inter-rater reliability study, many researchers wanted to know how to determine the optimal number of raters and the optimal number of subjects that should participate in the experiment. The fourth edition of this text addresses those needs, and more.
Features of the Fourth Edition include:

• New material on sample size calculations for chance-corrected agreement coefficients, as well as for intraclass correlation coefficients.
• The chapter entitled "Benchmarking Inter-Rater Reliability Coefficients" has been entirely rewritten.
• The introductory chapter has been substantially expanded to explore possible definitions of the notion of inter-rater reliability.
• All chapters have been revised to a large extent to improve their readability.

The primary objective of this book is to give researchers and students in all fields of research, the opportunity to access in one place, detailed, well-organized, and readable materials on inter-rater reliability assessment. Chance-corrected agreement coefficients are covered as well as agreement coefficients in the family of intraclass correlations. Other rank-based measures of association, and agreement coefficients within the framework of item analysis are discussed as well.

The methods and techniques developed in this handbook can handle missing ratings, which are common in most experiments. Errors found in this third edition are documented on the Error Page. Users of the statistical package R may download r functions for computing many of the coefficients discussed in this book.

Learn more about the content of this book, by downloading chapter descriptions below. For some chapters, the reader may also download Excel spreadsheets showing detailed calculations related to the chapter examples.

Chapter Number Description (Extract) Chapter Example (Excel Spreadsheet)
Chapter 1 Preliminaries & Chapter 1 N/A
Chapter 2 Rater Agreement for Nominal Ratings chapter2examples.xlsx
Chapter 3 Rater Agreement for Ordinal, Interval, Ratio ... chapter3examples.xlsx
Chapter 4 Constructing a Coefficient: AC1 & Aickin's Alpha chapter4examples.xlsx
Chapter 5 Agreement Coefficient: Statistical Inference N/A
Chapter 6 Benchmarking IRR N/A
Chapter 7 ICC: A Measure of Agreement N/A
Chapter 8 ICC: One-Factor Studies chapter8examples.xlsx
Chapter 9 ICC: Random Factorial Design chapter9examples.xlsx
Chapter 10 ICC: Mixed Factorial Design chapter10examples.xlsx
Chapter 11 IRR: Conditional Analysis chapter11examples.xlsx
Chapter 12 Measures of Association... chapter12examples.xlsx
Appendix B Appendix B (r functions) N/A


Purchase the book using one of the options below

Purchase the e-version of the Handbook and start reading immediately for $39.95

handbook of inter-rater reliability

Order a hard copy of the Handbook for $45.95

handbook of inter-rater reliability

Order the Handbook - Bundle - (Paperback & PDF eBook) for $59.95


Order from (US)

Order from (UK)

Order from (Canada)

Order from (Germany)

Order from (Japan)

Order from (Spain)

Order from (Italy)

Order from (France)


About the Author (K. L. Gwet, Ph.D) Contact Information
A statistical consultant, researcher and instructor. He has over 15 years of experience in various industries in the US and Canada. He possesses a doctorate degree in statistics from Carleton University - Canada -

Contact the author at,

  • (Note: remove all numbers 123 from email address. (i.e. @123agreestat becomes
  • Advanced Analytics, LLC
    PO Box 2696
    Gaithersburg, MD 20886-2696