human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

next stop

Robotics is bringing Betsy DeVos to Detroit for the first time as education secretary

Education Secretary Betsy DeVos. (U.S. Department of Education)

Betsy DeVos is set to appear in Detroit for the first time as education secretary on Friday, though she’s unlikely to encounter local students when she’s there.

DeVos is scheduled to attend a student robotics competition being held downtown in a bid to promote science and math education. The event is also likely to again highlight DeVos’s past influence over education policy in the city, which has been heavily scrutinized.

Before becoming President Trump’s education chief, DeVos, a prominent Michigan philanthropist, was a key architect of policies that many blame for the dire state of Detroit’s schools.

We’ve outlined that debate in full, but the key points are that the state’s charter law puts no restrictions on where or how many charter schools can open, which has created school deserts in some neighborhoods, and far too many schools in others. Both district and charter schools struggle financially with less-than-full enrollments, while student performance suffers across the board.

DeVos’ critics say she has blocked attempts to bring order and oversight to Detroit schools. Defenders note that parents now have more options and that charter school students in the city do slightly better on state exams than their peers in district schools.

DeVos also had a tense exchange with Lesley Stahl of “60 Minutes” about Michigan schools back in March.

“Michigan schools need to do better. There is no doubt about it,” she said.

DeVos’s announcement says she plans to meet with students on Friday. But while the event is happening in Detroit, the students DeVos encounters at the FIRST Robotics World Championship on Friday will almost surely hail from elsewhere. Earlier this week, Chalkbeat noted that just one city high school in Detroit qualified to send a team.

money talks

Funding for New York City homeless students, universal literacy in de Blasio’s executive budget

PHOTO: Benjamin Kanter/Mayoral Photo Office
New York City Mayor Bill de Blasio announces his 2019 budget proposal in the Blue Room at City Hall.

Mayor Bill de Blasio’s $90 billion city budget proposal includes millions of dollars for homeless students and to fuel a push to get every student reading on grade level by third grade.

The mayor’s official budget reveal comes after a major announcement Wednesday that the city will invest $125 million in schools, which principals can spend on items such as teacher salaries, after-school programs or new technology.

Taken together, the news means that New York City schools have avoided any budget cuts and instead received a sizeable boost in a year of funding uncertainty.

De Blasio took several shots at state lawmakers while unveiling his budget, emphasizing that the city invested in schools even as they received less than they anticipated in school funding from Albany.

“This certainly shows that even when Albany steps back, we step forward,” de Blasio said.  

Here’s what you need to know about education:

$30.5 million to boost literacy

De Blasio has said he wants his new schools chancellor, Richard Carranza, to “supercharge” his “universal literacy” program, which is attempting to help every third grade student read on grade level. On Thursday, de Blasio shed a little light on what he meant by outlining a plan to help the city’s neediest students.

The mayor’s plan would double after-school programs for students in shelters; provide more training for teachers of students learning English and students with disabilities; and boost the number of literacy coaches in low-performing schools.

De Blasio said that, though it hasn’t captured headlines, the city’s universal literacy program is going to be a focus for him moving forward. “This is one of the things the chancellor and I talked about the most during the interview process,” he said.

$12 million for social workers for students in shelters

The executive budget restores funding for homeless students that the preliminary budget lacked. For the past two years, de Blasio has left the funding stream out of his preliminary budget — drawing criticism from advocates.

That city’s budget will fund 53 social workers, according to Randi Levine, a policy director at Advocates for Children. Advocates have been calling for 150 social workers that would be spread out across schools and in shelters.

$23 million for anti-bias training

In the next school year, the city expects to train 10,000 education department employees, with the goal of reaching everyone in the department by the 2021-22 school year. The plan includes identifying schools that are adept in culturally relevant teaching so they can share their practices with other educators, and digging into data to uncover and address inequities in schools.

Derrick Owens, a father of two in Harlem, said he expects the expanded training will have a real impact in the classroom and help change the fact that students of color are disproportionately disciplined at school. Owens is a member of the Coalition for Educational Justice, a parent organization that has lobbied hard for more anti-bias training for teachers.

“Now what happens with the anti-bias training, teachers can identify a problem,” he said. “They won’t be quick to have the child disciplined or suspended. They’ll be able to work it out and able to solve the problem. I think it’s a win win.”

A “surprising” lack of funding from Albany

In his latest critique of Gov. Andrew Cuomo, de Blasio said the city expected to get $140 million in school aid from state lawmakers that never materialized. The state increased education spending by about $1 billion this year, but the boost was less than the city expected, de Blasio said.

“It was honestly very surprising that the number came in as low as it did,” de Blasio said.

Christina Veiga contributed reporting.