Jason Academy Summative Program Evaluation Final Report

Abstract

Method:

SRI conducted an evaluation study of JASON Academy courses to ascertain whether they are effective in increasing teachers’ content knowledge on topics that they must teach and that are part of most state standards and curriculum frameworks. SRI, in conjunction with JASON Academy staff, selected two JASON Academy courses for study, Structure of the Earth and Forces and Motion. These courses were selected because of the popularity of the courses and their clear connections to teaching standards. Students in one course served as a control for the other course.

SRI’s summative evaluation used a pre-post, comparison-group (quasi-experimental) design with teacher content knowledge as the chief outcome variable of interest. Because treatment and comparison groups were not matched, SRI used statistical controls to account for differences between groups in its analysis. Teacher surveys provided information on levels of participation; data from surveys were matched to assessment data to test the second hypothesis. Both the survey and content knowledge tests were developed by SRI. The content tests were aligned to content standards in the NSES. The surveys and assessments were administered to a total of 4 cohorts of students during the period from February 2, 2004 to November 19, 2004.

SRI, in conjunction with JASON Academy staff, prepared the pre- and post-course survey on teacher background, current teaching practices, and impact of the course on teaching practices. SRI crafted a pre- and post-course assessment for each course that focuses on conceptual understanding of core items in each course as identified in the NSES. The surveys and assessments were administered to a total of 4 cohorts of students.

Results:

Teachers made statistically significant gains in content knowledge, as compared to a control group, after taking the JASON Academy courses. Teachers self-reports indicated that many felt they had achieved their goals of gaining content knowledge and learning new ways to teach the science content as a result of taking the class. For neither of the courses was a correlation between self-reported activity level and gain on the assessment detected. Because the study did not randomly assign teachers to condition, these results cannot provide conclusive evidence that JASON Academy courses caused these gains; however, they do provide promising evidence of efficacy that could be investigated in future studies.

Our survey found that these teachers generally did not take the JASON Academy course in order to be able to incorporate more state or NSES content or process standards into their teaching, although most were familiar with the NSES and some reported feeling more prepared after taking the course. They instead reported goals of understanding science concepts better and gaining curriculum resources for their classroom. When asked to state how the course affected their approach to teaching science, the majority of teachers said that they learned new science content that they could in turn teach to their students. While many did not report feeling strongly that they were more prepared to incorporate NSES into their teaching after the course, they did feel more prepared to teach complex scientific ideas to their students. 43% of the participants said they “very much achieved” an improvement in their skills in incorporating technology into their teaching.