Seeking the Real Reliability: Why the Traditional Estimators of Reliability Usually Fail in Achievement Testing and Why the Deflation-Corrected Coefficients Could Be Better Options

Abstract

<p>Traditional estimators of reliability such as coefficients alpha, theta, omega, and rho (maximal reliability) are prone to give radical underestimates of reliability for the tests common when testing educational achievement. These tests are often structured by widely deviating item difficulties. This is a typical pattern where the traditional Pearson correlation between items and score (Rit) may be radically deflated. Because Rit is embedded in the traditional estimators of reliability, this causes deflation in the estimates of reliability, and the magnitude of deflation may be remarkable. Within achievement testing, deflation-corrected estimators of reliability (DCER) would be better options. Instead of Rit, DCERs use other estimators of correlation as the linking factor between the item and the score variable that are less prone to deflation. Selecting wisely the linking coefficient, DCERs may give significant advance in estimating the true reliability and true standard error related to the test score.</p><p>Keywords: reliability, deflation-corrected reliability, deflation in reliability, deflation in correlation, itemscore correlation, coefficient alpha, coefficient theta, coefficient omega, maximal reliability</p>