Arkansas

College and Career Readiness for All Students

"The higher the standards that are targeted, the clearer it becomes that the only way to take the students to those standards is early intervention and a long-term focus on improving the fundamentals of teaching and learning. Setting lower standards and targeting short-term incremental test score gains, on the other hand, often tempts educators into shortsighted 'quick-fix' practices."

—Chrys Dougherty, Lynn Mellor, and Nancy Smith, National Center for Educational Achievement

2009 NCEA Methodology for Identifying Just for the Kids Higher Performing Schools: Elementary School (Grade 3)

General Approach

The National Center for Educational Achievement's (NCEA) analysis included student test results from spring 2007, 2008, and 2009. For Grades 4 - 8, NCEA used an academic growth analysis that examines whether the students in each school perform above predicted based on their prior year's scores. Since students are not tested prior to third grade, NCEA looked at performance levels of students from third grade who have been continuously enrolled in the school or district for three years or more, rather than prior test scores. In all cases, student and school demographics were taken into account in determining whether a school's students performed above predicted. For elementary schools, rather than aggregating the results of two different models, NCEA identified separate lists of higher performing schools in third grade and fourth through sixth grades.

The commonly used statistical technique of Hierarchical Linear Modeling (HLM) was used in both analyses. This technique makes a statistical adjustment to take into account the greater uncertainty of school results measured for smaller groups of students. The analysis was conducted separately for each subject, so that schools are ranked separately for literacy and mathematics.

Schools with many students close to the ceiling on the state test can not be measured well on growth. For this reason, and to include schools whose students are the highest performing in absolute terms, NCEA added the top 5% of schools in the state to the list based on the percentage of students scoring at or above NCEA's college readiness target.

The steps in the analysis are as follows:

Step 1: Dataset creation

Merge each student's test results for each year in the analysis with the same student's fall enrollment records from the same year and the prior two school years. Use this information to separate out test results of students who were continuously enrolled at the same school or district for at least three years ("continuously enrolled students"). If the school's grade configuration does not allow for three years of continuous enrollment, as in the case of a school with only Grades 3-5, then the students must be enrolled in that school for the maximum number of years that the grade configuration will allow, and they must be in the same district for three years.

Step 2: Performance measure

For each subject, regress each student's converted scale score (z-score) in each grade and year on the following variables: the student's low-income and English Language Learner status; whether the student was African American or Hispanic; the percentage of continuously enrolled tested students in the student's grade; the percentage of continuously enrolled tested students who are low-income; English Language Learners, African American, or Hispanic; and whether the school was a magnet school.

HLM software is used to calculate, for each school in each third grade subject, a separate set of school effects by year and a combined school effect across years.

Step 3: Identification of eligible schools

Identify those schools that were eligible to be ranked for a given subject based on having an adequate number of students in the analysis in each grade and year in that subject. A school is eligible if it has no missing grades in the most recent school year and no more than the maximum allowable number of missing grades shown in the table below for the previous school year.

Number of grade-year combinations in the school in previous year 1-4 5-6
Maximum allowable number of missing grade-year combinations in the previous year 0 1

A grade is missing for a given school year if test score records were available for fewer than 10 continuously enrolled students.

Step 4: School rankings

Divide the eligible schools into four low-income groups (0-25, 26-50, 51-75, and 76-100% low-income students). Rank the schools within each of these low-income groups by their combined school effect across grades and years—the performance measure calculated in Step 2.

Step 5: School selection

Select schools that meet the following criteria:

  1. The school was in the top 10% of schools in its low-income group based on the combined school effect across grades and years, and the school had better-than-predicted performance in all years.
  2. If the school was in the top 5% of the state based on the percentage of students meeting NCEA's college readiness target. The Higher Performing Lists will inform users about which set of criteria was used to identify the school as higher performing.
  3. The school was not on the state's "school improvement list" based on the state accountability system in the current year.

A school meeting all of the requirements in Step 5 is deemed to be higher performing in the subject in question.