Title: Interpreting Kappa in Observational Research: Base rate Matters
Kappa is a popular agreement statistic used to estimate the accuracy of two or more observers. Kappa is a consistent estimator of accuracy only when the assumptions and contextual factors outlined by Cohen (1960) are met. In this presentation, the response of kappa to differing base rates of the observed event is examined.† Methods for interpreting the accuracy of observers are presented. The methods for interpreting kappa will be demonstrated using observational data. A review of software designed to analyze agreement data will be presented.†
1. †Agreement statistics
3. †Base rate
1. †Participants will be able to determine if a data space is mutually exclusive and†††† exhaustive.
2. †Participants will be able to calculate kappa from a two-by-two table using available software.
3. †Participants will be able to calculate the best estimate of the base rate of a code from a two-by-two table.
4. †Participants will be able to use the best estimate of the base rate and an obtained kappa to interpolate the accuracy of raters
5. †Participants will be able to justify to a reviewer the acceptance of a kappa below the criterion value of .6, if rater accuracy is sufficient.
P.I.ís, students, staff. The only prerequisite is a basic understanding of agreement coefficients like kappa.
Taylor is a doctoral student in Early Childhood Special Education. She received
her masters in Child and Family Development from the