light of day

City releases Teacher Data Reports — and a slew of caveats

When the Department of Education’s embargo of Teacher Data Reports details lifted at noon today, news organizations across the city rushed to make the data available.

The Teacher Data Reports are “value-added” assessments of teachers’ effectiveness that were produced from 2008 to 2010 for reading and math teachers in grades 3 to 8.

This morning, department officials including Chancellor Dennis Walcott and Chief Academic Officer Shael Polakow-Suransky met with reporters to offer caution about how the data reports should be used. They emphasized the reports’ wide margins of error — 35 percentage points for math teachers and 53 percentage points for reading teachers, on average — and that the reports reflect only a small portion of teachers’ work.

“We would never advise anyone — parent, reporter, principal, teacher — to draw a conclusion based on this score alone,” Polakow-Suransky said.

Most of the news organizations that filed Freedom of Information Law requests for the ratings plan to publish them in searchable or streamlined databases, with the teachers’ names attached. GothamSchools does not plan to publish the data with teachers’ names or identifying characteristics included because of concerns about the data’s reliability.

At least two other news organizations that cover education are also not publishing the data: the local affiliate of Fox News, according to a representative of Fox, and the nonprofit school information website Insideschools.

Department officials are asking schools not to release the reports to parents. They issued a guide today advising principals about how to handle parents who demand that their child be removed from the class of a teacher rated ineffective.

“Resist changing student/teacher assignments mid-year, as doing so is disruptive to all students’ learning,” the guide advises.

“We definitely will not be moving kids around based on a data point that is two years old,” Polakow-Suransky said today.

Joel Klein, who was chancellor when the reports were launched, was a public champion of the idea that releasing teacher ratings would empower parents to demand better schools and teachers for their children. He argued publicly that the ratings of individual teachers should be released after news organizations filed legal requests for them.

But Walcott began distancing himself from that position almost immediately after becoming chancellor last year. And last fall, his administration announced it had stopped producing the reports, citing a new state teacher evaluation system that will include value-added measures. Walcott explained in a Daily News column today that he is not thrilled that teachers “might be denigrated” as a result of the data dump.

Today, Walcott and Polakow-Suransky argued that Klein’s initial stance was justified — at the time. “What we’re looking at in 2012 is quite different from what Joel was looking at in 2010,” Polakow-Suransky said.

Because New York City adopted value-added measures relatively early, he said, the ratings were in fact among the most sophisticated measures available at the time. But even at the time, the ratings were not meant for public consumption or to be used in high-stakes decision-making, he said.

Polakow-Suransky acknowledged one exception to the high-stakes decision ban: School officials did use value-added ratings to inform decisions about whether to grant some teachers tenure.

We reported last year that principals said they were being prevented from granting tenure to teachers whose students had low test scores, particularly if they were among the roughly 12,000 teachers annually to receive a data report.

Polakow-Suransky said today that 133 teachers up for tenure last year were flagged because of low value-added scores. More than a third of the flagged teachers received tenure after their principals and superintendents determined that other factors carried more weight, he said.

The city released reports for about 18,000 different teachers over the course of the three school years during which they were produced. About 12,000 teachers received reports in each of the 2008-2009 and 2009-2010 school years, and about half of them received two because they taught both reading and math. (A smaller pilot group received reports in 2007-2008.)

Because many teachers’ students were tested in both math and English, some teachers were given two different value-added ratings for a single year — below average in one subject and above average in the other.

The reports also indicate whether teachers were found to have done especially well at boosting the test scores of particular types of students, such as students with disabilities, low-income students, or students who are considered English language learners.

Department officials said 77 percent of the teachers who received reports are still working in the city schools.

Next week, ratings will be released for teachers in schools for students with severe disabilities and for charter school teachers. The release of charter school teachers’ data reports reverses the department’s earlier decision not to release them because charter schools had opted into the Teacher Data Report program voluntarily.

Officials cautioned that the scores were especially unreliable for teachers with relatively few students. That group includes elementary school teachers, who typically had fewer than 30 students factor into their scores.

They also said reports are less reliable for teachers whose students were either all high-performing or mostly low-performing. The difference is due to the nature of the state tests, which were designed to distinguish among middle-level students — those just above and below the state’s proficiency cutoff. As a result, small differences among students at the top and bottom had an outsized impact on a teachers’ rating, Polakow-Suransky said.

The issue was one of several lessons learned that Polakow-Suransky said the city has flagged for the State Education Department, which is building its own value-added algorithm to be used as a part of new teacher evaluations.

The state is working with the American Institute of Research to build its model; the city’s model was designed by a research center at the University of Wisconsin.

Other lessons Polakow-Suransky identified include the importance of publishing value-added scores in the context of their margins of error and the need to create a way for teachers and principals to verify data.

The city introduced a verification system that teachers could use two years into the Teacher Data Report project. Less than 40 percent of teachers signed in to use it, but a significant number of them found major errors. For example, three percent of the teachers who signed in discovered that they had been marked as teaching classes that they had not actually taught.

In their lawsuit attempting to stop the ratings from being released, United Federation of Teachers officials identified more than 200 such errors. UFT President Michael Mulgrew said today that the errors showed that the city had not been respectful of teachers in developing or releasing the ratings.

“This was a complete debacle in terms of how the DOE has handled [the reports] and the mismanagement of the data inside of the system,” he said.

Asked whether he thought the state’s model would avoid the pitfalls the UFT identified in the city’s reports, Mulgrew said, “I’m not confident.”

But he said the state’s evaluation requirements are an improvement over the Teacher Data Reports because the new evaluations will showcase the value-added calculations alongside other measures of teacher quality.

study says...

In new study of school-district effectiveness, New York City falls just below national average

PHOTO: Stephanie Snyder

Each year, state test scores offer a snapshot of how much New York City students have learned. But they say little about how the city’s schools stack up against other districts’, in part because the raw scores largely reflect student demographics — wealthier districts tend to have higher scores.

Now, a major new analysis of several years of test scores from across the country provides a better way to judge and compare districts: Instead of looking at a single moment, it shows how well school systems help students grow their skills over time.

Based on that measure, New York City falls just below the middle of the pack: In the five years from third to eighth grade, its students collectively make about 4.6 grade levels of progress — landing New York in the 35th percentile of districts nationally. By contrast, Chicago students advance the equivalent of six grades within those five years, giving the district one of the highest growth rates in the country.

Still, New York is slightly above average when compared to other large districts with many students from low-income families. And it trounces the state’s other urban districts — including Yonkers, Syracuse, and Rochester, which have some of the nation’s worst growth rates.

“Among big poor districts, it’s better than average,” said Sean Reardon, the Stanford University researcher who conducted the analysis. “In the grand scheme, it’s pretty middle-of-the-road.”

Reardon’s analysis — based on 300 million standardized tests taken by students across more than 11,000 school districts from 2009 to 2015 — is the largest of its kind. It looks both at student proficiency on third-grade math and English tests (that is, what share of students earned a score deemed “proficient”) and student growth between grades three and eight (how much their scores improved over time). Reardon’s research was supported by several foundations, including the Bill and Melinda Gates Foundation, which also provides funding to Chalkbeat.

The analysis controls for the differences in tests across states and over time by converting scores into a common scale that measures growth in grade levels, making it possible to compare nearly every district in the country to one another. (It excludes New York’s scores from 2015 and some grades in 2014 because of the high number of students who boycotted the state tests those years. However, each district’s five-year growth rates is actually an average of its year-over-year growth, so Reardon was still able to calculate a five-year rate for New York.)

Experts generally prefer growth rates over proficiency as a way to evaluate school quality, since growth measures the progress students make in school rather than where they started. Even if a district enrolls many poor students who are less likely than their affluent peers to hit the “proficiency” benchmark, its schools can still help them advance at a rate comparable to or even better than schools filled with wealthier students.

“Growth is way better than achievement,” said Douglas Ready, an education and public policy professor at Teachers College, Columbia University. “We know low-income students start school behind — the question is what do school districts do with the kids they get?”

New York’s growth rate falls just below the national median of 4.8 grade levels. Among big districts, its students made gains similar to those in Dallas and Detroit, and greater than students in Los Angeles, Miami, and Indianapolis.

By contrast, Rochester ranks rock-bottom nationally. In that high-poverty district, where the median income among families with children in the public schools is $26,000, students advanced about three grade levels in five years. Yonkers’ $48,000 median income is much higher, yet its schools barely do better, with students moving just 3.5 grade levels. (Among New York City public-school parents, the median income is $42,000.)

Reardon emphasized that test scores provide an important but incomplete picture of student learning, and growth rates are an imperfect measure of school effectiveness since factors outside of the classroom also influence how much students learn over time.

Still, he argued that officials who rate schools and parents who choose them would do much better to look at a school’s growth rate over its average test scores. In fact, he said, a focus on growth rates could theoretically drive down socioeconomic segregation since higher-income parents might be willing to enroll their children in schools with many poor students and low overall test scores if the schools nonetheless had outstanding growth rates.

Ready, however, pointed out that even when schools and districts are highly effective at helping students make progress, they are still unlikely to close the yawning achievement gaps that separate most poor and wealthier students from the time they start school. Reardon came to the same conclusion.

“The large gaps in students’ academic skills between low- and higher-[socioeconomic status] districts are so large,” Reardon’s analysis says, “that even the highest growth rate in the country would be insufficient to close even half of the gap by eighth grade.”

In response to the analysis, New York City education department officials pointed to the National Assessment of Educational Progress, a standardized test taken by a representative sample of students in each state and certain districts, including New York. Only one other district among the country’s 10 largest cities performed better in reading and math than New York, which had the highest share of low-income students reach the proficient level on the reading test.

“Our schools are the strongest they’ve ever been, with record-high graduation and college enrollment rates, and improving state test scores,” said the district’s spokesman, Will Mantell.

change up

Just as Lower East Side integration plan takes off, superintendent who helped craft it steps down

PHOTO: Christina Veiga
Carry Chan, left, will become acting superintendent in District 1 when Daniella Phillips, right, leaves this month to join the central education department.

The longtime superintendent of the Manhattan community district where parents pushed for a plan to desegregate the local schools is stepping down just as the plan gets underway.

After a decade at the helm of District 1, which includes the Lower East Side and East Village, Superintendent Daniella Phillips is leaving to join the central education department, Chalkbeat has learned. During the yearslong campaign for an integration plan, Phillips acted as a liaison between parents and the education department, which finally approved a new admissions system for the district’s elementary schools this fall.

She will be replaced by Carry Chan, who has also played a role in the district’s diversity efforts as the interim head of a new Family Resource Center, an information hub to help district parents sort through their school options. Chan takes over as acting superintendent on Dec. 18.

The leadership change comes at a crucial time for the district, which also includes a portion of Chinatown. Parents are currently applying to elementary schools, marking the first admissions cycle under the new enrollment system. Under the system, schools give certain students admissions priority based on their economic status and other factors, with the goal of every elementary school enrolling share of disadvantaged students similar to the district average.

It will be up to the new superintendent to help schools recruit and welcome a greater mix of families, and to help steer parents towards a wider range of schools. Advocates hope the district can become a model for the city.

“There is a torch that needs to be carried in order to really, fully execute,” said Naomi Peña, president of the district’s parent council. “The next superintendent has to be a champion for the mission and the cause.”

During heated public meetings, Phillips tried to keep the peace while serving as a go-between for frustrated integration advocates and reluctant education department officials. The tensions sometimes boiled over, with advocates directing their anger at Phillips — though they were eventually won-over and endorsed the final integration plan.

In her new role, she will oversee school consolidations as part of the education department’s Office of School Design and Charter Partnerships. In District 1, Phillips helped steer three such mergers, which often involve combining small, low-performing schools with ones that are higher achieving.

“It has been such a joy and privilege to be District 1 superintendent for over 10 years, and I’m excited for this next chapter in the district and my career,” Phillips said in an emailed statement.

Chan is a former principal who launched the School for Global Leaders, a middle school that focuses on community service projects and offers Mandarin classes. Last year, she joined the education department’s Manhattan support center, where she helped schools form partnerships in order to learn from one another.

Since October, Chan has served as the interim director of District 1’s Family Resource Center, which is seen as an integral part of making the new diversity plan work. Families must apply for seats in the district’s elementary schools, which do not have attendance zones like other districts. The family center aims to arm families with more information about their options, in the hopes that they will consider schools they may not have previously.

“I think we’re all really passionate about this plan and we really want this to work,” Chan said. “Communication is the key, and being transparent with how we’re progressing with this work.”