Inter-Rater Reliability Using the SAS System
Written to help researchers and students compute inter-rater reliability coefficients in SAS. For categorical ratings, it covers chance-corrected agreement coefficients — Cohen's Kappa, Gwet's AC1/AC2, Krippendorff's alpha, and others. For quantitative ratings, it covers several variants of the Intraclass Correlation Coefficient (ICC).
Preliminaries & five chapters
Chance-corrected agreement coefficients (CAC)
SAS/IML libraries plus the individual chapter programs that call them.
Analysis of contingency tables, no missing ratings.
Analysis of raw ratings.
Analysis of the distribution of raters by subject and category.
Benchmarking agreement coefficients.
Paired t-test for testing the difference of agreement coefficients for statistical significance.
Weight functions for generating predefined sets of weights.
Basic program for computing Kappa from a contingency table.
Basic program for computing Kappa from raw scores.
Fixes the diagonal and imbalance problems.
Fixes the diagonal and imbalance problems, raw-score version.
The FREQ procedure with an unbalanced rating table.
Alternative agreement coefficients from a contingency table, using SAS/IML.
Unweighted analysis of raw ratings using agreecoeff3raw.sas.
Analyzing raw data using agreecoeff3dist.sas.
Weighted agreement coefficients based on ordinal weights, using agreecoeff3raw.sas.
Handling missing ratings with the MISSING option of the FREQ procedure.
Agreement coefficients for 2 raters with missing ratings — an alternative to FREQ.
Fleiss' generalized Kappa and Gwet's AC1 via the MAGREE.SAS macro.
Unweighted analysis of Table 3.1 ratings using agreecoeff3raw.sas.
Testing the difference between two agreement coefficients for statistical significance.
Cumulative membership probabilities for benchmarking agreement coefficients.
Intraclass correlation coefficients (ICC)
SAS/IML libraries plus the Chapter 4 programs that call them.
Intraclass correlation coefficients for one-factor models.
Intraclass correlation coefficients for two-factor models.
Using the INTRACC.SAS macro and Shrout & Fleiss (1979), Table 2.
ICC under the one-factor ANOVA model.
ICC under the two-way ANOVA model with random effects (Model 2).
ICC under the two-way ANOVA model with mixed effects (Model 3).
Finn's agreement coefficient and its associated p-value.