Methodology - Texas Monthly's Best Public Middle Schools - Texas

2007 NCEA Methodology for Texas Monthly's Best Public Middle Schools

General Approach

NCEA's analysis looked at student test results from spring 2005, 2006, and 2007. For middle and high schools, NCEA used an academic growth analysis that examined whether the students in each school performed "above predicted" based on their prior year's scores. For elementary schools, no prior test scores were available for third grade, so NCEA looked at achievement levels of students from third grade and higher who had been continuously enrolled in the school for three years or more.¹ In all cases, student and school demographics were taken into account in determining whether a school's students performed above predicted.

The commonly used statistical technique of Hierarchical Linear Modeling (HLM) was used in both analyses. This technique makes a statistical adjustment to take into account the greater uncertainty of school results measured for smaller groups of students. The analysis was conducted separately for each subject, so that schools were ranked separately for reading/English Language Arts, mathematics, writing, science, and social studies.²

Schools with many students close to the ceiling on the TAKS test can't be measured well on growth. For this reason, and to include schools whose students are the highest-performing in absolute terms, we added to the list the top 5% of schools in the state based on the percentage of students scoring at or above NCEA's college readiness benchmark in reading, mathematics, or science, or the state's Commended standard in writing or social studies.

The steps in the analysis were as follows:

Elementary schools were defined as any school whose highest grade was six or lower. For schools with whose highest grade was seven or higher and whose lowest grade was five or below, the portion of the school ending in grade 5 was treated as an elementary school.

Step 1: (Dataset creation)

Merge each student's current year test results with those of the same student from the previous year. Current year math is merged with prior year math, reading with prior year reading, writing and social studies with prior year reading, and science with prior year math. Grade 11 English Language Arts (ELA), science, and social studies are matched with Grade 10 results in the same subject in the previous year, while Grade 10 ELA was matched with ninth grade reading. Scores from sixth grade reading and mathematics tests taken in Spanish were not used.

Current year fall enrollment records were also used to identify whether the student was enrolled at the same school during the current school year. Students without prior year scores in the relevant subject, or who were not enrolled in the same school in the fall of the current year, were not included in the analysis.

Step 2: (Performance measure)

For each subject, regress each student's score in each grade and year on the following variables: the student's prior year score in the relevant subject; the student's low-income and English Language Learner status; whether the student was African American or Hispanic; the student's grade level; the percentage of continuously enrolled tested students in the student's grade; the schoolwide percentages of low-income, English Language Learner (for middle schools only), African American, and Hispanic students; and whether the school was a magnet school. We also added a variable that allowed the relationship between prior and current year scores to vary close to the ceiling of the prior year test.

Hierarchical Linear Modeling (HLM) software was used to calculate, for each school in each subject, a separate set of "grade-and-year school effects" and a "combined school effect across grades and years."

Step 3: (Consistency measure)

For each school and subject, calculate the standard deviation of the separate grade-and-year school effects and use that standard deviation as a measure of consistency. Schools with a wide variation in performance across grades and years (less consistency) will have a large standard deviation, whereas schools with less variation across grades and years (more consistency) will have a low standard deviation.

Step 4: (Identification of eligible schools)

Identify those schools that were eligible to be ranked for a given subject based on their having an adequate number of students in the analysis in each grade and year in that subject. A school was eligible if it had no missing grades in the most recent school year (2006-07), and no more than the maximum allowable number of missing grades shown in the table below for the 2004-05 and 2005-06 school years.

Number of grade-year combinations in the school in 2004-05 and 2005-06³	2-4	5-6
Maximum allowable number of missing grade-year combinations in 2004-05 and 2005-06	0	1

A grade was missing for a given school year if either a) test score records were available for fewer than ten continuously enrolled students; or b) looking at 2007 records in particular, more than 20% of the records in that grade were deleted based on masking rules newly applied to student assessment data in that year.⁴

Step 5: (School rankings)

Divide the eligible schools into four low-income groups (0-25, 25-50, 50-75, and 75-100% low-income students). Rank the schools within each of these low-income groups by their combined school effect across grades and years - the performance measure calculated in Step 2. Rank the schools within each quartile separately by the consistency measure calculated in Step 3, with schools with lower standard deviations (more consistency) ranked higher.

Step 6: (School selection)

Select schools that meet all of the following criteria:

The school was in the top 10% of schools in its low-income group based on the combined school effect across grades and years.
The school was in the top half of its low-income group based on consistency (low variation in performance across grades and years).
If the school didn't meet criteria (1) and (2) but was in the top 5% of the state based on the percentage of students meeting NCEA's college readiness benchmark, then the school was treated as if it had met criteria (1) and (2).
In the case of reading/English Language Arts and mathematics, the school met federal Adequate Yearly Progress (AYP) requirements in 2007.
In the case of writing, science, and social studies, at least 85% of the students enrolled in each tested grade in 2007 took the TAKS test.⁵
The school was Academically Acceptable or better based on the state accountability system in 2007.

A school meeting all of the requirements in Step 6 was deemed to be consistently higher performing in the subject in question.

¹ Schools without tested grades (e.g., a PK-2 school), or elementary schools for which some tested grades would lack three-year continuously enrolled students (e.g., third or fourth graders in a grades 3-5 school) were not rated in this analysis.   back
² Middle schools were not ranked on science, because only two years of Grade 8 science test data were available as of spring 2007.   back
³ For example, a Grades 6-8 middle school in operation in both the 2004-05 and 2005-06 school years could have data from six grade-year combinations: Grades 6, 7, and 8 for each of 2004-05 and 2005-06.   back
⁴ The state legislature has established three Education Research Centers whose databases are not subject to these masking rules. Once the databases of these three centers are populated, then a more complete set of data will be available for analysis.   back
⁵ Many students in special education are eligible to take the State-Developed Alternative Assessment (SDAA), and some students' Individual Education Plans may specify that they take that test. This rule says that no more than 15% of the students should miss taking the TAKS, either because they are absent or because they take the SDAA test instead.   back

National Center for Educational Achievement / Just for the Kids

Texas

Texas Affiliate