Two hot-air balloonists get lost, and they’re floating aimlessly. They spot someone down below them, and call out, “Hello!” The person on the ground replies, “Hello!” “Where are we?” one calls down. Up comes the reply: “You’re in a balloon!” They continue to drift, and one of the balloonists says to the other, “Who was that?” And the other responds, “That was obviously an economist.” “An economist? How can you tell?” the first asked. “Because what he said was precise, but irrelevant,” the other replied.
“Precise, but irrelevant” is my three-word assessment of the recent study of traditional and alternative teacher certification conducted by Mathematica Policy Research for the Institute of Education Sciences in the U.S. Department of Education. (And the study really isn’t very precise, but that’s a more technical story.) The design of this study successfully precludes it from addressing the most salient policy questions about alternative teacher certification–but we get a pretty clean estimate of the relative effectiveness of pairs of traditional-route and alternate-route teachers that are not representative of any population of teacher education programs, teachers, or schools.
The biggest weakness of the study, in skoolboy’s opinion, is that it fails to take seriously the idea that the elements of teacher education programs differ from one another, and that there is variability in the quality of programs–within both the population of traditional teacher certification programs and the population of alternative route teacher certification programs. The design of the Mathematica study doesn’t evaluate the operations and outcomes of particular traditional or alternative programs. And yet most of the relevant policy questions pertain to investment in particular programs or the hiring of graduates of particular programs. The study design cannot address these questions.
Here’s what the Mathematica researchers did: They compiled a list of 165 alternative teacher certification programs across the country that are not as selective as high-profile programs such as Teach for America, some sponsored by institutions of higher education and others by school districts or regional education agencies, and drew a stratified random sample of 63 programs. Then they looked for elementary schools that had hired an alternatively-certified teacher from one of these programs within the past three to five years. They then filtered these schools to the subset that had hired a relative novice traditionally-certified teacher in the same grade, creating a pair of teachers-one traditionally certified, and one alternatively certified, both with less than five years of experience, in the same grade in the same school. Students in that grade were then randomly assigned either to the traditionally-certified teacher’s classroom or the alternatively-certified teacher’s classroom. The use of random assignment created what Mathematica refers to as a mini-experiment, which led the researchers to attribute any achievement differences observed at the end of the school year to the effect of having an alternatively-certified vs. traditionally-certified teacher. The researchers observed each teacher, rating them on classroom instruction practices, and distinguished between alternative-certified teachers from high-coursework programs and those from low-coursework programs. The resulting study involved 2,600 students in 63 schools in 20 districts, spread across 7 states. The key conclusion: “The study found no benefit, on average, to student achievement from placing an [alternatively-certified] teacher in the classroom when the alternative was a [traditionally-certified] teacher, but there was no evidence of harm, either.”
Sounds pretty good, right? No way to address the fact that prospective teachers self-select into traditional or alternative certification programs, of course, but we can’t rely on random assignment to solve that problem, and the researchers acknowledge this, saying, “Because of likely differences in the types of people who attend various certification programs, the results cannot be used to rigorously address how a graduate of one type of program would fare if he or she had attended another type.” A large, diverse sample … random assignment to control selection into a teacher’s classroom … what’s to complain about?
How about this: the alternatively-certified teachers and traditionally-certified teachers in the study were not necessarily representative of graduates of the programs they attended. In fact, both the alternatively-certified teachers and traditionally-certified teachers in the study were persisters; any program graduate who had left teaching in the early career–and there are concerns about early attrition within the alternate-route population–would not appear in the sample. Moreover, the alternative certification programs and traditional certification programs in the study were not necessarily representative of all alternative certification programs and traditional certification programs. The schools in the study were not necessarily representative of all schools that hire both alternatively-certified and traditionally-certified teachers. And finally, the skills and competencies appearing on the California Achievement Test (CAT-5) may not be aligned with, and hence representative of, state and district curricular priorities.
Some evidence of the problems that this causes: One-half of all of the teachers in the study are from a single state, Texas. 71% of all of the classrooms in the study are kindergarten through second grade. The average number of students per classroom at the start of the year was 15.
So: if you’re a Texas principal interested in hiring an early elementary grade teacher with a few years of experience into a small classroom based on generic standardized test scores, this is the study for you. On the other hand, if you are interested in the quality of particular traditional certification programs or alternative certification programs, at either the elementary or secondary levels … or if you are a policymaker wondering whether to increase investment in alternative certification programs … or if you’re a principal wondering whether your next hire should be a traditionally-certified or alternatively-certified teacher … keep looking. The Mathematica study won’t tell you what you want to know.
About our First Person series:
First Person is where Chalkbeat features personal essays by educators, students, parents, and others trying to improve public education. Read our submission guidelines here.