Calculating Intraclass Correlation with AgreeStat 2011.1 Posted : October 24, 2011

AgreeStat 2011.1 for Excel Windows provides the simplest way for researchers to compute the Intraclass Correlation Coefficient (ICC). It is a self-automated workbook containing a Visual Basic for
Applications program, and requires no installation. You simply need to have MS Excel 2007 or 2010 for Windows. The Mac version of AgreeStat 2011.1 will be released in the beginning of 2012. Download the trial version of AgreeStat 2011.1 here, and you will be surprised to see how easy and intuitive it is.

Numerous versions of the intraclass correlation coefficient are offered for the purpose of evaluating the extent of agreement among multiple raters based on interval or ratio data. Here are 3 key advantages for using AgreeStat 2011.1 to compute the intraclass correlation coefficients:

• AgreeStat 2011.1 is based on the familiar user-friendly Excel environment within which you can manipulate your data with ease.
• AgreeStat 2011.1 implements the most popular intraclass correlation coefficients introduced by Shrout and Fleiss (1979).
• Unlike the existing statistical packages (including SPSS, SAS, and others), AgreeStat 2011.1 can handle missing values with no problem. Records with a few missing scores are not removed from analysis. Instead, non-missing from those records are used in the calculation if the intraclass correlation coefficient as suggested by Searle (1997). This results in a more efficient use of your statical data, and a more accurate evaluation of the extent of agreement among raters.
• In addition to computing inter-rater reliability coefficients based on intraclass correlation, AgreeStat 2011.1 also allows you to compute intra-rater reliability coefficients, which are commonly used by researchers.

Figure 1 shows AgreeStat 2011.1's main menu, which allows to specify a 2-rater or 3-rater analysis. Although this post focuses on intraclass correlation, AgreeStat 2011.1 allows you to compute chancecorrected agreement coefficients such as Cohen's kappa, Fleiss' generalized kappa, Gwet's AC₁, Scott's π, and many more, along with their standard errors.

Figure 2 shows how you would capture your data for the purpose of calculating the intraclass correlation coefficient of your choice. You would first select your workbook, then the worksheet within your workbook, before specifying the data itself.

Figure 3 shows the "Options (ICC)" form that allows you to specify the model you want to use for calculating the intraclass correlation coefficient. Depending on the model of your choice, you would calculate Shrout-Fleiss' ICC(1,1), ICC(1,k), ICC(2,1), ICC(2,k), ICC(3,1), or ICC(3,k) coefficients (see Shrout and Fleiss, 1979 for more information regarding these coefficients.

Figure 1: AgreeStat's Main Menu

Figure 2: Capturing Data for the Calculation of Intraclass Correlation Coefficients

Figure 3: Specifying Inter-Rater Reliability Model

 

References

• Shrout, P.E., and Fleiss, J.L. (1979). "Intraclass Correlations: Uses in Assessing Rater Reliability." Psychological Bulletin, 86, 420-428.
• Searle, S.R. (1997). Linear Models (Wiley Classics Library), Wiley-Interscience: John Wiley & Sons, Inc. (see pages 473- 493)
  

Back to the Inter-Rater Reliability Discussion Corner's Home Page

---

Please use the form shown below to communicate your comments, questions, or suggestions to me regarding this post. I will be happy to get back to you as soon as possible. Thanks. K.L. Gwet, Ph.D.

Subject*:
Name* . :
E-Mail* .:

Comments:

 

Submit Your Message: