Lombardi, A. Seburn, M., Conley, D., & Snow, E. (2010). A generalizability investigation of cognitive demand and rigor ratings of items and standards in an alignment study. Paper presented at the 2010 annual meeting of the American Educational Research Association (AERA), Denver, CO.
Introduction
In alignment studies, expert raters evaluate assessment items against standards and ratings are used to compute various alignment indices. Questions about rater reliability, however, are often ignored or inadequately addressed. This paper reports the results of a generalizability theory study of cognitive demand and rigor ratings of assessment items and college-readiness standards in the context of an alignment study of college admissions tests to a set of college readiness standards. Results indicate a higher level of generalizability for Math item and standard ratings than for English item and standard ratings, as well as a higher level of generalizability for cognitive demand ratings than for rigor ratings. Results also suggest that the standard of 5-6 raters in alignment studies may be insufficient for obtaining desired reliability. These findings may be used to carefully plan more robust alignment studies in the future so that higher levels of reliability across raters will be attained. (Contains 3 tables and 2 figures.)
Keywords: Generalizability Theory, Test Items, College Entrance Examinations, Readiness, Academic Standards, Cognitive Processes, Difficulty Level, Reliability, Alignment (Education), Mathematics Tests, Language Tests, English