Methodology - Best Practices - NCEA/Just for the Kids

Best Practice Methodology

History of the NCEA Best Practice Research

After Just for the Kids (JFTK) posted the first Just for the Kids School Reports, educators began asking for more information about the schools in their Top Comparable Schools lists. They wanted to know, in essence, "What are these schools doing that we are not?" JFTK began studying the practices of Higher Performing Texas Schools in 1999. The early investigations of these schools were intentionally not based in theory. Instead, JFTK leadership made a conscious decision to visit Higher Performing Schools (identified on the basis of their performance on the state criterion-referenced test) and simply share what they were doing that was different from average-performing schools.

The study methodology has evolved continuously over the past five years. Although the original "grounded" studies (1999-2001) provided a foundation upon which the more recent investigations rest, the study scope and methods have been tightened since the formation of the National Center for Educational Achievement in 2001. JFTK state affiliates from Washington and Colorado joined Texas in investigating the practices of high-performing schools that year.

In the fall of 2003, state affiliates from Arkansas, California, Illinois, and Tennessee joined Colorado, Texas, and Washington to establish best practice research studies under the banner of NCEA State-Comparable Best Practice Studies. Studies continued during the 2004-05 school year in Arkansas, California, Colorado, Illinois, Tennessee, Texas, and Washington, with new studies initiated in five new states: Florida, Hawaii, New Jersey, New York, and Oklahoma. More than 500 schools across the country have been studied by NCEA or affiliated state-based researchers, creating a substantial database of schools in a variety of policy and community contexts.

By sharing a common methodology, the studies in these states were designed to enable results to be both relevant in the state context and more easily compared across states for a national view of what successful schools were doing to support student achievement.

Study Goals and Research Questions

Whereas the methodology has developed over the years, the goal of the NCEA Best Practice Research remains the same-to illuminate and describe the key differences between what Higher Performing Schools and average-performing schools do to support student achievement. In particular, these studies are focused on schools that are "beating the odds" relative to student performance.

The primary research question addressed by the NCEA State Best Practice Studies is "How do Higher Performing Schools in the state differ from average-performing schools?" Particular supplemental questions may also be posed in conducting these studies including "How do the educational structures differ between the two types of schools?"

Yin (2003) positions such research questions in the case study tradition. "In general, case studies are the preferred strategy when 'how' or 'why' questions are being posed, when the investigator has little control over events, and when the focus is on a contemporary phenomenon within some real-life context" (Yin 2003, p. 1). Certainly, the NCEA State-Comparable Best Practice Studies meet these criteria. Although the current federal research trend favors experimental research conducted to establish causal links, well-executed case study research serves a valuable purpose in illuminating possible correlations and promising areas for random-trial research.

State-based Research Teams

Since 2001-02, state-based research teams have participated in NCEA's effort to study schools in a variety of state policy contexts in order to bring greater expertise of state-specific issues to the project. These teams were usually university-based, but some state efforts were led by other non-profit groups. Following are the research groups that have been involved with this study at some point during the past six years:

Arkansas: National Office for Research, Measurement and Evaluation Systems (NORMES) at University of Arkansas at Fayetteville
California: Springboard Schools on behalf of Just for the Kids-California
Colorado: Research and Development Center for the Advancement of Student Learning at Colorado State University
Florida: Council for Educational Change
Hawaii: Collaborative Action for Public Education (CAPE)
Illinois: College of Education at Illinois State University
New Jersey: Graduate School of Education at Rutgers University
New York: University at Albany School of Education
Oklahoma: Oklahoma Commission for Educational Leadership
Tennessee: Tennessee State University
Texas: University of Texas at Austin
Washington: Washington School Research Center at Seattle Pacific University

Methodology Overview

NCEA Best Practice Studies follow typical qualitative methods of data collection and analysis. Site visits to identified schools include interviews with district and school administrators, as well as with a number of classroom teachers. Supporting district or school documents are collected, and a series of self-audit surveys is administered to some district and school staff.

Site Selection

A rigorous school identification process is fundamental to the NCEA Best Practice Study research methodology. School selection for a state study is based primarily on the school's student achievement data from the state assessment, which spans three years in order to measure consistency. The same process is used in every NCEA best practice study, although modifications may be required in some states, depending based on the availability of student achievement data.

Under consideration in the identification of consistent Higher Performers are (1) the school's performance on the state's assessment over a period of three years, in all tested grades and subjects, (2) the demographic characteristics of the student body, including the percentage of economically disadvantaged students, and (3) the students' preparation for the school level (for example, the average scores of students the year prior to entering high school) whenever longitudinal data is available. The regression analysis itself compares the performance of each school, in each tested grade and subject, to the predicted performance of demographically similar schools. Additional information about the identification of Higher Performing Schools is located under Consistently Higher Performing Schools.

"Average-performing" schools are identified the same way, with their performance consistently placing them around the average or predicted value for similar schools. Including average-performing schools in the study is an important way of establishing that the findings actually distinguish the Higher Performing Schools in important ways.

Data Collection and Analysis

The identified school is the primary unit of analysis, but the entire setting of the school is explored, through interviews and document collection at the district, school, and classroom levels. Site visits to identified Higher Performing Schools and average-performing schools include interviews with key district leaders, school leadership, and classroom teachers. Most interviews occur individually, although schedules often require that teachers be interviewed in focus group settings. Documents such as policy statements, curriculum documents, lesson plans, professional development offerings, master schedules, etc., are collected and reviewed as they relate to the operations of the target school.

The interview data is coded and analyzed using standard techniques for qualitative analysis. Usually, more than one researcher codes each school's data to provide a measure of inter-rater reliability. From the coded interviews and document review, researchers prepare a case study for each school (both Higher Performing and average-performing).

Comparison of higher and average-performing school cases points to practices of Higher Performing Schools that differ, either in quantity or quality from average-performing schools.

Assumptions and Limitations

The NCEA Best Practice Research staff is especially interested in studying schools that have "beaten the odds" in terms of student performance; schools that meet this selection criteria are of even greater interest to our research. For this reason, the identification criteria may permit schools into the pool of consideration that are of little interest when compared to schools that are successfully addressing the needs of traditionally underserved, disadvantaged student populations.

Schools are the primary unit of analysis, but the methodology also includes interviews at the district and classroom levels, in consideration of the entire context in which the school operates. Interview data at various levels of the school system, as well as supporting documentation, provide opportunities for data triangulation.

The majority of the data collected in these studies is self-reported. It is assumed that informants will be truthful in their responses. The methodology relies on a variety of interview participants and supporting documents to help underscore the accuracy of self-reported data.

At each site, a relatively small number of personnel may be interviewed. Administering the NCEA Self-Audits to all school staff helps maximize the number of voices heard on a site visit. While these data are also self-reported, the study is able to reflect the perceptions of the widest possible range of informants.

The structure of the NCEA State Best Practice Research Program assumes that site visits to Higher Performing Schools and average-performing schools will yield enough information to allow a coherent and meaningful picture of the school system's operations.

National Center for Educational Achievement / Just for the Kids Find your school: OR %
About Us Staff Partners Our Supporters Careers Donate Contact Us Just for the Kids Inform Inspire Improve History State Partners Research & Policy College/Career Readiness Higher Performing Schools Best Practices Longitudinal Data Teacher Preparation Broad Prize 2003-2005 Data Quality Campaign Tools & Services District & School Services Press Room News Publications Presentations Events
College/Career Readiness Higher Performing Schools Identification Methodology Identification Research Best Practices Search for Evidence Case Studies Methodology Executive Summaries About the Framework The Themes The Organizational Levels The Practices Longitudinal Data Teacher Preparation Data Collection Completed Tasks Contacts Broad Prize 2003-2005 Data Quality Campaign	Best Practice Methodology History of the NCEA Best Practice Research After Just for the Kids (JFTK) posted the first Just for the Kids School Reports, educators began asking for more information about the schools in their Top Comparable Schools lists. They wanted to know, in essence, "What are these schools doing that we are not?" JFTK began studying the practices of Higher Performing Texas Schools in 1999. The early investigations of these schools were intentionally not based in theory. Instead, JFTK leadership made a conscious decision to visit Higher Performing Schools (identified on the basis of their performance on the state criterion-referenced test) and simply share what they were doing that was different from average-performing schools. The study methodology has evolved continuously over the past five years. Although the original "grounded" studies (1999-2001) provided a foundation upon which the more recent investigations rest, the study scope and methods have been tightened since the formation of the National Center for Educational Achievement in 2001. JFTK state affiliates from Washington and Colorado joined Texas in investigating the practices of high-performing schools that year. In the fall of 2003, state affiliates from Arkansas, California, Illinois, and Tennessee joined Colorado, Texas, and Washington to establish best practice research studies under the banner of NCEA State-Comparable Best Practice Studies. Studies continued during the 2004-05 school year in Arkansas, California, Colorado, Illinois, Tennessee, Texas, and Washington, with new studies initiated in five new states: Florida, Hawaii, New Jersey, New York, and Oklahoma. More than 500 schools across the country have been studied by NCEA or affiliated state-based researchers, creating a substantial database of schools in a variety of policy and community contexts. By sharing a common methodology, the studies in these states were designed to enable results to be both relevant in the state context and more easily compared across states for a national view of what successful schools were doing to support student achievement. Study Goals and Research Questions Whereas the methodology has developed over the years, the goal of the NCEA Best Practice Research remains the same-to illuminate and describe the key differences between what Higher Performing Schools and average-performing schools do to support student achievement. In particular, these studies are focused on schools that are "beating the odds" relative to student performance. The primary research question addressed by the NCEA State Best Practice Studies is "How do Higher Performing Schools in the state differ from average-performing schools?" Particular supplemental questions may also be posed in conducting these studies including "How do the educational structures differ between the two types of schools?" Yin (2003) positions such research questions in the case study tradition. "In general, case studies are the preferred strategy when 'how' or 'why' questions are being posed, when the investigator has little control over events, and when the focus is on a contemporary phenomenon within some real-life context" (Yin 2003, p. 1). Certainly, the NCEA State-Comparable Best Practice Studies meet these criteria. Although the current federal research trend favors experimental research conducted to establish causal links, well-executed case study research serves a valuable purpose in illuminating possible correlations and promising areas for random-trial research. State-based Research Teams Since 2001-02, state-based research teams have participated in NCEA's effort to study schools in a variety of state policy contexts in order to bring greater expertise of state-specific issues to the project. These teams were usually university-based, but some state efforts were led by other non-profit groups. Following are the research groups that have been involved with this study at some point during the past six years: Arkansas: National Office for Research, Measurement and Evaluation Systems (NORMES) at University of Arkansas at Fayetteville California: Springboard Schools on behalf of Just for the Kids-California Colorado: Research and Development Center for the Advancement of Student Learning at Colorado State University Florida: Council for Educational Change Hawaii: Collaborative Action for Public Education (CAPE) Illinois: College of Education at Illinois State University New Jersey: Graduate School of Education at Rutgers University New York: University at Albany School of Education Oklahoma: Oklahoma Commission for Educational Leadership Tennessee: Tennessee State University Texas: University of Texas at Austin Washington: Washington School Research Center at Seattle Pacific University Methodology Overview NCEA Best Practice Studies follow typical qualitative methods of data collection and analysis. Site visits to identified schools include interviews with district and school administrators, as well as with a number of classroom teachers. Supporting district or school documents are collected, and a series of self-audit surveys is administered to some district and school staff. Site Selection A rigorous school identification process is fundamental to the NCEA Best Practice Study research methodology. School selection for a state study is based primarily on the school's student achievement data from the state assessment, which spans three years in order to measure consistency. The same process is used in every NCEA best practice study, although modifications may be required in some states, depending based on the availability of student achievement data. Under consideration in the identification of consistent Higher Performers are (1) the school's performance on the state's assessment over a period of three years, in all tested grades and subjects, (2) the demographic characteristics of the student body, including the percentage of economically disadvantaged students, and (3) the students' preparation for the school level (for example, the average scores of students the year prior to entering high school) whenever longitudinal data is available. The regression analysis itself compares the performance of each school, in each tested grade and subject, to the predicted performance of demographically similar schools. Additional information about the identification of Higher Performing Schools is located under Consistently Higher Performing Schools. "Average-performing" schools are identified the same way, with their performance consistently placing them around the average or predicted value for similar schools. Including average-performing schools in the study is an important way of establishing that the findings actually distinguish the Higher Performing Schools in important ways. Data Collection and Analysis The identified school is the primary unit of analysis, but the entire setting of the school is explored, through interviews and document collection at the district, school, and classroom levels. Site visits to identified Higher Performing Schools and average-performing schools include interviews with key district leaders, school leadership, and classroom teachers. Most interviews occur individually, although schedules often require that teachers be interviewed in focus group settings. Documents such as policy statements, curriculum documents, lesson plans, professional development offerings, master schedules, etc., are collected and reviewed as they relate to the operations of the target school. The interview data is coded and analyzed using standard techniques for qualitative analysis. Usually, more than one researcher codes each school's data to provide a measure of inter-rater reliability. From the coded interviews and document review, researchers prepare a case study for each school (both Higher Performing and average-performing). Comparison of higher and average-performing school cases points to practices of Higher Performing Schools that differ, either in quantity or quality from average-performing schools. Assumptions and Limitations The NCEA Best Practice Research staff is especially interested in studying schools that have "beaten the odds" in terms of student performance; schools that meet this selection criteria are of even greater interest to our research. For this reason, the identification criteria may permit schools into the pool of consideration that are of little interest when compared to schools that are successfully addressing the needs of traditionally underserved, disadvantaged student populations. Schools are the primary unit of analysis, but the methodology also includes interviews at the district and classroom levels, in consideration of the entire context in which the school operates. Interview data at various levels of the school system, as well as supporting documentation, provide opportunities for data triangulation. The majority of the data collected in these studies is self-reported. It is assumed that informants will be truthful in their responses. The methodology relies on a variety of interview participants and supporting documents to help underscore the accuracy of self-reported data. At each site, a relatively small number of personnel may be interviewed. Administering the NCEA Self-Audits to all school staff helps maximize the number of voices heard on a site visit. While these data are also self-reported, the study is able to reflect the perceptions of the widest possible range of informants. The structure of the NCEA State Best Practice Research Program assumes that site visits to Higher Performing Schools and average-performing schools will yield enough information to allow a coherent and meaningful picture of the school system's operations.

National Center for Educational Achievement / Just for the Kids