IXL contracted with SRI Education for an independent review of a quasi-experimental study they conducted: “Assessing the Impact of IXL Math over Three Years: A Quasi-Experimental Study.” The study author conducted a retrospective analysis to investigate the extent to which school use of IXL Math during a three-year period was associated with higher rates of proficiency on statewide assessments. SRI determined that this study provides moderate evidence for IXL Math efficacy in grades 3-8 over a three-year period according to the ESSA levels of evidence provided by the U.S. Department of Education guidelines.
Xin Wei
SRI Author
Evaluation of Los Angeles City College’s STEM Pathways Program: Impacts of STEM Pathways Program Participation on Student Outcomes
Los Angeles City College launched the STEM Pathways program in 2016 with funding from the U.S. Department of Education. The intent of the STEM Pathways program was to improve STEM degree completion and transfer to 4-year colleges, particularly for low-income and Latinx students. The program offered a variety of supports including Supplemental Instruction (SI), peer tutoring in STEM, a book and technology loan program, specialized counseling, a math boot camp, and an undergraduate research experience. This report presents findings from a quasi-experimental study estimating the impact of STEM Pathways program participation on STEM course success and continuation in STEM, as well as degree attainment. SRI found positive and statistically significant results for continuation in STEM and STEM credits earned.
Evaluation of Los Angeles City College’s STEM Pathways Program: Impacts of the Supplemental Instruction Program on Student Outcomes
Los Angeles City College launched the STEM Pathways program in 2016 with funding from the U.S. Department of Education. The intent of the STEM Pathways program was to improve students’ STEM degree completion and transfer to 4-year colleges, particularly for low-income and Latinx students. The program provided a variety of supports including Supplemental Instruction (SI) for key STEM courses. This report presents findings from a quasi-experimental study estimating the impact of SI participation on STEM course success and continuation in STEM. SRI found positive and statistically significant results for focal course passing, focal course grade, and continuation in STEM.
Evaluation of Los Angeles City College’s STEM Pathways Program: Impacts of the STEM Learning Center on Student Outcomes
Los Angeles City College launched the STEM Pathways program in 2016 with funding from the U.S. Department of Education. The intent of the STEM Pathways program was to improve students’ STEM degree completion and transfer to 4-year colleges, particularly for low-income and Latinx students, through a variety of supports. The STEM Learning Center, a drop-in peer tutoring program offering students assistance in STEM courses, was one of the grant’s most-used supports. This report presents findings from a quasi-experimental study to estimate the impact of the STEM Learning Center on STEM course success and continuation in STEM. SRI found positive effects of STEM Learning Center participation on students’ STEM outcomes, in terms of increased STEM credits and STEM continuation.
Measuring the Impact of Trauma-Focused, Cognitive Behavioral Group Therapy with Middle School Students
This study examines differential effects of the Cognitive Behavioral Intervention for Trauma in Schools (CBITS) program on behavioral and academic outcomes of middle school students. Researchers administered screenings to grade 6 students to assess traumatic stress and then randomized those with elevated levels to the CBITS treatment ( n = 150) or comparison group ( n = 143). Analyses examined the overall impact of CBITS and differential effects among subpopulations of students who reported clinically significant externalizing ( n = 73) or internalizing behavior ( n = 181) at baseline. Results demonstrated that relative to counterparts in the comparison group, externalizers in CBITS reported significantly reduced post-traumatic stress, anxiety, anger, internalizing and externalizing problems, and total behavior problems, and they also significantly improved scores on a standardized literacy assessment at posttest. Students with internalizing behavior problems showed differential academic effects at 1-year follow-up; those in CBITS did significantly better on standardized math tests but significantly worse on a reading subtest than their counterparts.
Innovations in Early Mathematics: Final Report
SRI’s evaluation of a 2-year professional development (PD) intervention for PreK to 3 rd grade teachers found that high-quality PD focusing on teachers’ instructional practices and pedagogical content knowledge, as well as their dispositions toward math, can lead to significant changes in teacher behavior across preK and early elementary grades.
In partnership with the Erikson Institute Early Math Collaborative, SRI evaluated the impact of teachers’ participation in a 2-year professional development (PD) intervention to improve teachers’ knowledge, attitudes, and instructional practice in math across prekindergarten (preK) to third grade classrooms in the city of Chicago. Compared to teachers receiving business as usual PD, participation in Erikson’s Whole-Teacher PD led to robust changes in teacher practices after 1 year and were sustained at the 4-year follow-up observations. Intervention teachers also reported higher confidence in their ability to use high-quality math instructional practices after completing the 2-year PD compared to business-as-usual teachers. Limited impacts on student learning were observed. The findings suggest high-quality PD focusing on teachers’ instructional practices and pedagogical content knowledge, as well as their dispositions toward math, can lead to significant changes in teacher behavior across preK and early elementary grades.
Promoting science, technology, and engineering self-efficacy and knowledge for all with an autism Inclusion maker program
This paper describes the collaborative development of an inclusive maker program called Inventing, Designing, and Engineering for All Students (IDEAS) and the results of a study on the impact of that program on autistic students and their neurotypical peers. The IDEAS project brought together experts in maker education, autism inclusion, engineering, co-design, and research. Over 2 years, this group adapted and pilot tested a museum-based maker program so that it could be run as an informal club in autism-inclusion middle schools (students ages 10–14) in New York City. in the United States. In the third year, teachers in each school implemented the redesigned program on their own. Researchers conducted a mixed-methods study of the impact of the program on participants. The study used observations and interviews; social interaction analysis, a pre/post survey of science, technology, and engineering self-efficacy and career interest; and a pre/post assessment of understanding of the engineering design process (EDP). Autistic and neurotypical students were in either the treatment condition (if they joined the maker program) or a business-as-usual comparison condition (if they did not join the club). Our analyses of the survey and EDP assessment compared the maker group with the comparison group and showed that participating in the maker program led to improved outcomes in the following constructs: technology and engineering self-efficacy (effect size = 0.80), technology and engineering interest (effect size = 1.73), vicarious experience (effect size = 0.57), science appreciation (effect size = 0.21), and understanding of the engineering design process (effect size = 0.44). The maker program benefited neurotypical students more than autistic students on technology and engineering interest and science appreciation, possibly because autistic students started with a high level of STEM interest. Qualitative analysis demonstrated that all students engaged in the EDP and pursued a wide range of interests, that autistic students who struggled in normal school settings were successful in creating their projects and communicating with peers about those projects, and that teachers reported being better able to see what their autistic students were capable of accomplishing when they were freed from the constraints of typical classroom instruction.
ST Math: Nonregulatory ESSA Standards Evidence Review & What Works Clearinghouse Standards Review
MIND Research Institute contracted with SRI Education for an independent review of a quasi-experimental study WestEd conducted in partnership with MIND: “A Cross-State Evaluation of MIND Research Institute’s ST Math Program and Math Performance.” The study authors conducted multiple analyses to investigate the extent to which use of Spatial-Temporal (ST) Math in grades 3–5 was associated with increased school mathematics achievement.
SRI determined that this study provides moderate evidence for ST Math efficacy in grades 3–5 according to Every Student Success Act (ESSA) levels of evidence provided by the U.S. Department of Education guidelines. SRI determined that study fulfills the design, analytic, and technical requirements of WWC v4.0 Group Design Standards for Meets Evidence Standards with Reservations.
The authors selected two outcomes of interest for study: average scale scores and percent proficient rates on state standardized mathematics assessments. The authors applied multiple comparison corrections for their main findings. On both outcomes, grade-level clusters using ST Math performed significantly better in mathematics than similar grade-level clusters not using the program. As measured by average scale scores on state standardized mathematics assessments, the authors found a positive effect size of .13. As measured by percent proficient rates on state standardized mathematics assessments, the authors found a positive effect size of .17. Considerably larger positive effects were found in reduced analytic samples of grade-level clusters implementing ST Math according to specific fidelity metrics.
Student Outcomes in Health Pathways
This brief presents findings from the Oakland Health Pathways Project (OHPP), a joint initiative of Oakland Unified School District, Alameda Health System, and Alameda County Health Care Services Agency. The initiative is designed to improve educational and long-term employment outcomes for youth of color in Oakland (Alameda County), California, while expanding and diversifying the local health care workforce. It applies Linked Learning, an approach to college and career preparation that combines classroom learning with real-world work experiences. This brief compares health pathway students’ outcomes with those of students who had similar demographic characteristics and prior achievement but were in two other programs of study: 1) traditional high school programs, and 2) other career-themed pathways. The study found health pathway students significantly outperformed traditional high school students on the high school course credits earned, high school graduation, college eligibility requirements met, and college enrollment. There were no differences in health pathway student outcomes in high school English Language Arts and math achievement, and dropout rates when compared to students in traditional high school programs and other career-themed pathways. In addition, health pathway students were no different in completion of college eligibility requirements and, among those who enrolled in college, were no more likely to enroll in a 4-year (rather than 2-year) college when compared to students in traditional high school programs and in other career-themed pathways.
