human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

that was weird

The D.C. school system had a pitch-perfect response after John Oliver made #DCPublicSchools trend on Twitter

Public education got some unexpected attention Sunday night when John Oliver asked viewers watching the Emmys to make #DCPublicSchools trend on Twitter.

Oliver had been inspired by comedian Dave Chappelle, who shouted out the school system he attended before he announced an award winner. Within a minute of Oliver’s request, the hashtag was officially trending.

Most of the tweets had nothing to do with schools in Washington, D.C.

Here are a few that did, starting with this pitch-perfect one from the official D.C. Public Schools account:

Oliver’s surreal challenge was far from the first time that the late-show host has made education a centerpiece of his comedy — over time, he has pilloried standardized testing, school segregation, and charter schools.

Nor was it the first education hashtag to take center stage at an awards show: #PublicSchoolProud, which emerged as a response to new U.S. Education Secretary Betsy DeVos, got a shoutout during the Oscars in February.

And it also is not the first time this year that D.C. schools have gotten a surprise burst of attention. The Oscars were just a week after DeVos drew fire for criticizing the teachers she met during her first school visit as secretary — to a D.C. public school.

Startup Support

Diverse charter schools in New York City to get boost from Walton money

PHOTO: John Bartelstone
Students at Brooklyn Prospect Charter School in 2012. The school is one of several New York City charters that aim to enroll diverse student bodies.

The Walton Family Foundation, the philanthropy governed by the family behind Walmart, pledged Tuesday to invest $2.2 million over the next two years in new charter schools in New York City that aim to be socioeconomically diverse.

Officials from the foundation expect the initiative to support the start of about seven mixed-income charter schools, which will be able to use the money to pay for anything from building space to teachers to technology.

The effort reflects a growing interest in New York and beyond in establishing charter schools that enroll students from a mix of backgrounds, which research suggests can benefit students and is considered one remedy to school segregation.

“We are excited to help educators and leaders on the front lines of solving one of today’s most pressing education challenges,” Marc Sternberg, the foundation’s K-12 education director and a former New York City education department official, said in a statement.

Walton has been a major charter school backer, pouring more than $407 million into hundreds of those schools over the past two decades. In New York, the foundation has helped fund more than 100 new charter schools. (Walton also supports Chalkbeat; read about our funding here.)

Some studies have found that black and Hispanic students in charter schools are more likely to attend predominantly nonwhite schools than their peers in traditional schools, partly because charter schools tend to be located in urban areas and are often established specifically to serve low-income students of color. In New York City, one report found that 90 percent of charter schools in 2010 were “intensely segregated,” meaning fewer than 10 percent of their students were white.

However, more recently, a small but rising number of charter schools has started to take steps to recruit and enroll a more diverse student body. Often, they do this by drawing in applicants from larger geographic areas than traditional schools can and by adjusting their admissions lotteries to reserve seats for particular groups, such as low-income students or residents of nearby housing projects.

Founded in 2014, the national Diverse Charter Schools Coalition now includes more than 100 schools in more than a dozen states. Nine New York City charter groups are part of the coalition, ranging from individual schools like Community Roots Charter School in Brooklyn to larger networks, including six Success Academy schools.

“There’s been a real shift in the charter school movement to think about how they address the issue of segregation,” said Halley Potter, a senior fellow at the Century Foundation, a think tank that promotes socioeconomic diversity.

The Century Foundation and researchers at Teachers College at Columbia University and Temple University will receive additional funding from Walton to study diverse charter schools, with the universities’ researchers conducting what Walton says is the first peer-reviewed study of those schools’ impact on student learning.