Without some correction, the percentage of agreement between two observers will tend to overestimate the true level of interrater reliability.

Question

Quizplus · Accepted Answer

This is because the observers may agree on certain aspects of the observation by chance, leading to a higher percentage of agreement that does not necessarily reflect true interrater reliability. Correction methods such as Cohen's kappa or intraclass correlation coefficients can provide a more accurate estimate of interrater reliability.

Without Some Correction, the Percentage of Agreement Between Two Observers