Gauging Misclassification in Rapid Guessing Identification in a Fast-Paced Vocabulary Test

Abstract

<p>In low-stakes testing, rapid-guessing behavior (RG) presents a significant challenge to the validity of test scores. This study investigates the misclassification produced by nine different response time (RT) threshold methods in identifying RG, using large-scale assessment data from a fast-paced vocabulary test and introducing choice reaction time (CRT) as a ground truth variable. Although the methods varied mostly in their ability to estimate thresholds at all and not in the misclassification rates nor in their nature, the results show significant misclassification rates across methods, ranging from .080 to .096 (Finnish speakers) and from .087 to .164 (non-Finnish speakers). All methods were more conservative than liberal, with false negatives outnumbering false positives. The findings emphasize the problem of the binary mind-set in RG identification, and suggest that there is a need for approaches that identify RG at the participant-by-item level in order to improve the accuracy of RG identification.<br></p>