Charting A Course: Evaluation Design Of The National School District And Network Grants Program


Smerdon, B., Garet, M., O’Day, J., Miller, L., Means, B., & Mitchell, K. (2003, April). Charting a course: Evaluation design of the National School District and Network Grants Program. Paper presented at the annual meeting of the American Educational Research Association, Chicago.


Over the past decade, a number of reform efforts have aimed at reducing the size of the learning communities in the nationā€™s schools. Most recently, the Bill & Melinda Gates Foundation has committed more than $375 million to improving Americaā€™s high schools through its National School District and Network Grants Program. Likely to be one of ā€œthe most publicly scrutinized educational initiativesā€ in recent history, the foundationā€™s program is providing a catalyst to educators and policy makers to radically reshape secondary education in the United States. The foundation is working with school districts and network organizations to create education systems that work for all students, including those students who have been characterized in the past as most underserved by the nationā€™s education system. At the secondary level, the foundation is supporting small schools as a corrective to the large, impersonal ā€œshopping mallā€ high schools that dominate today. To this end, the foundation supports: 1) districts and network organizations in converting large high schools into smaller learning communities, and 2) network organizationsā€™ efforts to replicate successful model schools or to implement small, effective schools from the ground up.

Through a competitive award process, the foundation selected our organizationsā€”the American Institutes for Research (AIR) and SRI Internationalā€”to evaluate its initiative. We are currently beginning our third year of data collection. The purpose of this paper is to describe the framework underlying our evaluation design of a large-scale, multifaceted school reform initiative and to identify some of the methodological challenges inherent in work of this kind. Throughout this paper, we use travel as a metaphor for our evaluation design process. Specifically, we begin by describing the earliest stages of our journeyā€”our starting point. Here we outline the foundationā€™s theory of changeā€”their goals and vision for the initiative. We then turn to a discussion of defining the destinationā€”the research questions we seek to answerā€”and finding the coordinatesā€”the conceptual framework that will guide our evaluation. Next, we explore the terrain, providing an overview of the grantees and schools in the population, and determine our route (e.g., the design basics of our evaluation). We conclude with a discussion section that we have entitled, Reading the Map While Driving. Here we discuss a few of the challenges of our evaluation and our need for reflection and refinement as the initiative grows and changes and as we learn through our research activities.

Read more from SRI