Title: Interpreting Kappa in Observational Research: Base rate Matters

 

 

Abstract

Kappa is a popular agreement statistic used to estimate the accuracy of two or more observers. Kappa is a consistent estimator of accuracy only when the assumptions and contextual factors outlined by Cohen (1960) are met. In this presentation, the response of kappa to differing base rates of the observed event is examined.Methods for interpreting the accuracy of observers are presented. The methods for interpreting kappa will be demonstrated using observational data. A review of software designed to analyze agreement data will be presented.

 

 

Topics

1. Agreement statistics

2. Kappa

3. Base rate

4. Accuracy

 

Goals

1. Participants will be able to determine if a data space is mutually exclusive and†††† exhaustive.

2. Participants will be able to calculate kappa from a two-by-two table using available software.

3. Participants will be able to calculate the best estimate of the base rate of a code from a two-by-two table.

4. Participants will be able to use the best estimate of the base rate and an obtained kappa to interpolate the accuracy of raters

5. Participants will be able to justify to a reviewer the acceptance of a kappa below the criterion value of .6, if rater accuracy is sufficient.

 

Intended Audience:

P.I.ís, students, staff. The only prerequisite is a basic understanding of agreement coefficients like kappa.

 

 

Speaker Description

Cornelia Taylor is a doctoral student in Early Childhood Special Education. She received her masters in Child and Family Development from the University of Georgia. While at Vanderbilt, she has worked with Dr. Paul Yoder on several projects investigating the validity and reliability of observational measures of young children with disabilities. Both classical and modern test theory have been used in these projects. She is currently completing her dissertation project comparing children with ASD to typically developing children on the probability of correctly responding to items on a parent report vocabulary inventory controlling for expressive and receptive vocabulary.††††††