|
Author: Dr. John Cadena, Director of Testing and Evaluation, San Antonio Independent School District (SAISD)
Relation of author to model: District evaluator – This work was commission by the SAISD superintendent for a report to the school board.
Date evaluation report completed: Spring, 2000
Months/years covered by evaluation: Spring 96 through Spring 99—three data points (inclusive of baseline) predicting implementation in Spring 99. Actual year of implementation varied by school. Nearly all middle schools in the study were in their second year of implementation, for elementary schools, most were in their third year.
Number of schools involved: 64 elementary schools and 17 middle schools. Nine of the elementary and six of the middle schools were implementing MRSH.
Demographics of sample population: At year-end, 1999-2000, the total number of students in the San Antonio Independent School District (SAISD) was 55,704 with the Elementary School population composed of 29,628 students and 12,617 students in Middle Schools. SAISD’s ethnic composition consisted of: 85% Hispanic, 10% Black (not Hispanic), 4% White (not Hispanic), and less than 1% Asian or Pacific Islander, or American Indian. The Pupil-Teacher ratio was approximately 16 to 1. Seventeen percent of the students were Limited English Proficient, and 79% were economically disadvantaged.
Purpose of study: To compare student performance at schools implementing a CSR model in SAISD. Effects were estimated for four designs and then for “non” participating schools.
Methodology: To evaluate the relative effectiveness of each design, three data points were used from each elementary and middle school: mathematics pass rate, reading pass rate, and writing pass rate. While schools that chose to implement a design tended to be among the lowest performing schools in the district, there was variation in the starting points. Thus, the analysis sought to find the achievement effects independent of baseline achievement. Similarly, the proportion of new teachers was not distributed evenly across schools, requiring that one evaluate the effects of the designs independent of the proportion of new teachers in the building.
Design effects were evaluated, then, by predicting 98-99 pass rates for all middle and elementary schools, with three data points per school. Predictors of that pass rate were the pass rates in previous years (95-96, 96-97, and 97-98), design affiliation (Success for All, Co-Nect, Modern Red SchoolHouse, Expeditionary Learning, or none), and the proportion of inexperienced teachers. Predicted means were calculated for each design and compared with the actual mean pass rates to establish an “effect size” for each of the designs, independent of prior achievement levels and the proportion of new teachers in 1998-99. Figure 3 shows the coefficients for various design team effects, independent of those factors, benchmarked against the “effect size” of the unaffiliated schools.
Measures/indicators used to assess effectiveness: Students in Texas are required to take the Texas Assessment of Academic Skills (TAAS) in grades three through eight. Reading and mathematics skills are assessed at each of these grades. Writing is evaluated at the fourth and eighth grades only. School-wide pass rates are provided for each subject and were used in this analysis. Thus, each school had three data points used in the analysis: school-wide pass rates in mathematics, reading, and writing.
Figure 3
San Antonio Independent School District, San Antonio, Texas:
Analysis of CSR design effects on TAAS* pass rates** in reading,
grades 3-8, Spring 1999
|
 |
* Texas Assessment of Academic Skills
** Pass rate in 1999 regressed on pass rates in previous years
(1997, 1998), years of implementation, proportion of new teachers
 Deviation from Predicted Score**
|
Findings: The results show substantial differences in effects across designs. The Modern Red SchoolHouse design has an effect three times greater than any other design implementing in the district. The MRSH coefficient (or deviation from predicted score) is shown as the fourth bar of Figure 3. Other designs’ coefficients are shown on the first, second, and third bars of Figure 3. While one design (shown in the first bar) failed to even meet the expected pass rates, given their baseline scores, and two designs were modestly above their expected pass rates, MRSH pass rates are measurably higher than their predicted rates. In essence, it says that pass rates were, on average, 1.6% higher than expected for MRSH schools. The effect size for MRSH is also greater than for those schools not affiliated with a comprehensive school reform—typically the higher achieving schools in SAISD. The coefficient for those schools is shown as the fifth bar. |
