Agreement Among 3 Raters or More When a Subject Can Be Rated by No More Than 2 Raters

Most multi-rater agreement methods assume every rater scores every subject. This post describes a design that relaxes that assumption — useful when having all raters see all subjects is too expensive or too demanding to be realistic.

The motivating example is laboratory accreditation: verifying a lab's procedures and documentation is expensive, so an accrediting body may only be willing to fund one round of rating per lab, with a second rating coming from the lab's own regular accreditation process. That means any given lab is really only ever seen by two raters — never by the full roster of raters the agency might use across all its accreditation work.

The question the post asks is whether it's still possible to quantify agreement across a larger group of raters — say, five — when no single subject is ever rated by more than two of them. The answer is yes, through what's called a partially-crossed design with two raters per subject.

How the design works

1. List every possible pair that can be formed from the full roster of raters. With 5 raters, that's 10 distinct pairs.
2. For each subject in the study, randomly select one of those pairs (with replacement) to do the rating, rather than assigning the same fixed pair to everyone.
3. Once all subjects are rated, the resulting dataset looks just like a standard two-rater dataset — a subject column and two rating columns — and the usual agreement coefficient formula still applies directly to it.

The catch is precision, not validity: because which two raters see a given subject is now itself random, that extra layer of randomness increases the variance of the resulting agreement coefficient compared to a fully-crossed design where the same two raters see every subject. The coefficient is still a valid measure of agreement across the whole rater roster — it's just a somewhat less precise one, and that trade-off is the price of making the design logistically feasible at all.