Citation
DeLyser, L. A., & Rachmatullah, A. (2026). A principled design of educational AI benchmarks. SRI.
Abstract
Generative artificial intelligence (AI) is increasingly used to support instruction, assessment, tutoring, educational data analysis, and administrative work, often without rigorous evidence of what AI systems can do reliably or where their performance falls short. This white paper argues that educational AI benchmarks can complement studies of educational impact and human–AI interaction by providing repeatable evidence about capabilities demonstrated by AI systems under specified conditions. Drawing on Principled Assessment Design, we propose a preliminary Principled Benchmark Design framework that guides developers in defining targeted capabilities, identifying observable evidence, designing tasks and scoring procedures, testing the benchmark, documenting appropriate score interpretations, and maintaining the benchmark as technologies and uses evolve. We illustrate the framework through three use cases: generating first-grade science assessments, extracting and redacting information from Individualized Education Program documents, and applying a pedagogical content knowledge rubric to science lesson plans. These examples demonstrate how benchmark design can address instructional, administrative, and evaluative applications of AI while making underlying claims, evidence, conditions, and limitations explicit. The framework emphasizes that benchmark scores describe system performance under documented evaluation conditions and should not be interpreted as blanket certifications of quality, effectiveness, or safety.


