Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities

This project uses innovative statistical and machine learning techniques to understand the differences in test-taking processes of students with and without disabilities.

Researchers from SRI, in partnership with colleagues from the University of Illinois, Urbana-Champaign, received funding from the Institute of Education Sciences to examine National Assessment of Education Progress (NAEP) process, outcome, and survey data to understand the test-taking behavior and mathematics performance of learners with and without disabilities.

The NAEP process data record the series of clicks, entries, and timestamps during test takers’ interactions with the test items. By analyzing process data, the SRI research team can examine the key components of test-taking behavior of learners with disabilities and uncover patterns that would not be apparent if using outcome or survey data alone. We will use innovative machine learning approaches coupled with expert review to:

• Extract meaningful patterns from process data to reflect key components of students’ mathematics test-taking behavior (i.e., math cognitive process, time on task, level of engagement, and accommodation usage).
• Examine how test-taking behavior differs by students’ disability status.
• Identify patterns of test-taking behavior that lead to successful performance on the NAEP mathematics grade 8 assessment.
• Investigate the association between test-taking behaviors and test performance for students with disabilities and students without disabilities.

In addition, the SRI research team will use structural equation modeling (SEM) to understand the degree to which the interrelationships among math instruction, performance, and key components of testing behavior (i.e., math cognitive process, time on task, level of engagement, and accommodation usage) differ by students’ disability status.

The findings of this study will provide important insights about the test taking behavior of students with disabilities and their cognitive processes, time on task, level of engagement, and accommodation usage during their interactions with a digital assessment. Results may also reveal helpful strategies that students with disabilities can use to solve math problems, and features of digital assessments that increase engagement and accessibility and provide more equitable and accurate measures of the performance of students with different abilities.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324P210005 to SRI International. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.