Title:
Interpreting Kappa in Observational Research: Base rate Matters
Abstract
Kappa is a
popular agreement statistic used to estimate the accuracy of two or more
observers. Kappa is a consistent estimator of accuracy only when the
assumptions and contextual factors outlined by Cohen (1960) are met. In this
presentation, the response of kappa to differing base rates of the observed
event is examined. Methods for
interpreting the accuracy of observers are presented. The methods for
interpreting kappa will be demonstrated using observational data. A review of
software designed to analyze agreement data will be presented.
Topics
1.
Agreement statistics
2.
Kappa
3.
Base rate
4.
Accuracy
Goals
1. Participants will be able to determine if a data
space is mutually exclusive and
exhaustive.
2. Participants will be able to calculate kappa
from a two-by-two table using available software.
3. Participants will be able to calculate the
best estimate of the base rate of a code from a two-by-two table.
4. Participants will be able to use the best
estimate of the base rate and an obtained kappa to interpolate the accuracy of
raters
5. Participants will be able to justify to a
reviewer the acceptance of a kappa below the criterion value of .6, if rater accuracy
is sufficient.
Intended
Audience:
P.I.’s,
students, staff. The only prerequisite is a basic understanding of agreement
coefficients like kappa.
Speaker
Description
Cornelia
Taylor is a doctoral student in Early Childhood Special Education. She received
her masters in Child and Family Development from the