With funding from the Bill & Melinda Gates Foundation, SRI International is conducting a study investigating to what extent, why, when, and how online learning models that target Algebra 1 work or do not work for different student populations, especially those that are historically disadvantaged or underserved.
Adoption of online learning models by students, teachers, schools, districts, and states has outpaced the research base on how to implement online learning most effectively for various student populations. As a result, decision-makers and policymakers have little objective, research-based evidence to inform their choice of one product over another, or how online learning might best be employed. As evidence suggesting the effectiveness of online learning accumulates, the policy dialogue needs to move to how to maximize achievement in online environments.
In this study, SRI will develop evidence-based models and tools to support the implementation of online learning that maximizes student success, particularly of at-risk students. The emphasis of the work is on features rather than individual products. Our approach is not to rank courses or providers, but to identify strategies that look across products to identify generalizable design and implementation principles.
- What is the range of design and implementation affordances of online Algebra 1 courses currently available to educators and students? What strategies are being used by online providers to support the success of all students, especially at-risk students, in online courses?
- How effective are design and implementation strategies in encouraging student participation, completion, and academic achievement, while considering factors such as learning time and cost?
- In estimating likely success of programs prospectively in the future, what elements are most important? How should they be measured?
Research Methods and Reporting
SRI will take a holistic, empirical approach to the identification of high-quality online learning features and practices associated with academic success and greater college readiness through systematic investigation of (a) student academic outcomes; (b) design, function, and use variability; and (c) costs.
The study began with a literature review, synthesizing research to compile a list of features that may be related to online learning quality and student success—features that have high face validity, are considered best practices, have expert support, and/or have some empirical validation—such as pedagogical approach, opportunities for student reflection, immediacy and type of feedback, and policies that lower barriers to participation. These features in turn formed the basis of a courseware review protocol that will help us determine whether and how providers incorporate them into their products.
Also during Phase 1 of the research, we developed the “Provider Profiles.” To develop the Provider Profiles, 6 providers of online Algebra 1 operating at sufficient scale were purposefully selected from an initial list formed from a scan of resources. These profiles offer systematic, comparable findings regarding the range of evidence-based principals derived from the literature and collected from provider websites, interviews with providers, and independent review of provider courseware. The profiles also serve the immediate purpose of providing educational decision makers with summaries of the various online Algebra 1 products currently on the market.
During Phase 2, we will use system use and outcomes data to empirically validate whether or not the features identified in Phase 1 are associated with student success. We will work the providers featured in the Phase 1 profiles to obtain student demographic and outcomes data including participation, course completion, final course grades, and standardized test scores such as state-mandated end-of-course exam scores (where available). We will also request log files from the providers so that we may determine students’ interaction with and actions within the online course environment.
The models developed in Phase 2 will provide evidence to consumers, developers, and researchers regarding the effectiveness of practices used to address students’ needs in online Algebra 1 and identify generalizable design and implementation principles that can be used to estimate the likely success of programs in the future.
“Success” will be defined across multiple dimensions, including both traditional definitions such as course completion, participation rates, and standardized measures of academic learning (i.e., state EOC exams, standardized general math assessments), as well as more emergent measures such as growth and learning rate (early course completion or wider scope of learning in a set amount of time).
Data Analytic Strategy
We will apply educational data mining and learner analytics methods to datasets supplied by participating providers to explore possible relationships between program features and student success. The implementation and use data will inform the frequency and patterns of use of particular tools or affordances within a course. Associations between design and implementation features, as well as student outcomes will be examined via multiple regression and its extensions.
Variables in the model will include those with evidence-based support in the literature, as well as distinctive features identified during the independent content review or by providers. For example, in addition to student demographic variables, we will consider design features (e.g., learner control), online teacher practices (e.g., rapid response, individualized instruction), and vendor support practices (e.g., technical assistance, tutors). If possible, we will also view student engagement with the courseware as both a potential outcome measure (examining the correlations between engagement and features) and as a moderator of student learning.
To prevent larger providers from “swamping” the results and to address comparability challenges with measures across providers (in terms of metrics, scales, etc.), we will take a meta-analytic approach to the Phase 2 work; analyses will be conducted within provider and patterns will be identified across providers.