Federal review questions conclusiveness of principals study

A federal review is challenging the results of a study that showed graduates of New York City’s principal academy outperformed their peers in some areas.

But one of the study’s authors says the reviewers misunderstood his work.

The reviewers argued that because Leadership Academy principals were frequently placed in more challenging schools than their counterparts in the study, differences in outcome could be due to other factors besides the principal’s training.

The review was completed by the What Works Clearinghouse, a program funded by the U.S. Department of Education’s Institute for Educational Science and charged with judging whether education researchers’ conclusions are warranted by their evidence.

Sean Corcoran, an assistant professor at New York University and one of the study’s authors, said that it was “not surprising” that the report did not meet WWC’s evidence standards because the organization’s guidelines are designed to judge a different style of research than the one he completed.

Corcoran also criticized the review for misstating the study’s original conclusions. The review cites researchers’ conclusions that elementary and middle schools with Leadership Academy principals saw no statistically significant difference in students’ English and math scores. In fact, the study reported a statistically significant change in students’ English, but not math, scores.

When the study was released, Chancellor Joel Klein highlighted it as evidence of the successes of his policies. But some of Klein’s critics have pointed out that study’s findings were mixed, and did not address many questions about the success of Leadership Academy graduates.

The research firm that runs the review program, Mathematica, had been commissioned by the New York City Department of Education to do a long-term study of the Leadership Academy; that study was canceled before the NYU researchers published their study.

The WWC completes two kinds of evaluations of education research. Roughly a third of the research given “quick reviews,” as this one was, are found not consistent with the body’s evidence standards, said Russ Whitehurst, the former director of the USDOE’s Institute for Education Sciences and now a fellow at the Brookings Institution.

Here’s the whole quick review of the study: