human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

state of the union

New York City teachers union braces for Supreme Court ruling that could drain money and members

PHOTO: Patrick Wall
UFT President Michael Mulgrew (standing) met with teachers during a school visit in 2014.

A few dozen labor leaders gathered recently at the the headquarters of New York City’s 187,000-member teachers union to hear a cautionary tale.

In a glass-walled conference room overlooking downtown Manhattan, United Federation of Teachers President Michael Mulgrew settled into a chair facing a colleague from Wisconsin. He asked the state teachers union president, Kim Kohlhaas, how her members have fared after an aggressive rollback of labor’s bargaining power there.

She described rampant teacher turnover, fewer job protections, and ballooning insurance and pension costs. In short, a union’s worst nightmare.

For the UFT, Wisconsin is a harbinger of what could result from a Supreme Court case known as Janus, which revolves around the ability of public unions to collect mandatory fees. Oral arguments begin on Feb. 26, and the decision, which is expected in a matter of months, could dramatically alter the landscape for unions across the country.

The impact will be felt especially by the UFT, the largest union local in the country. If the court rules that teachers are not required to pay for its services, the union is likely to shed members and money — a war chest that has allowed the UFT to be a major player in New York politics and to secure robust benefits for its members.

“This is dangerous stuff we’re getting into now,” Mulgrew told Chalkbeat. “They’re trying to take away people’s ability to come together, to stand up and have a voice.”

While the case deals with different issues than Wisconsin’s anti-union policies did, New York City labor leaders say the limits on their membership and funding would weaken their ability to fight against further restrictions on their organizing and bargaining power.

In anticipation of the ruling, union leaders have reportedly already considered downsizing their operations. And they have undertaken a preemptive information and recruitment campaign to hold onto members — who, soon, may be free to choose whether to keep supporting the union financially.

“Much as I oppose Janus, it’s kind of a wake up call for entrenched union leadership,” New York City teacher Arthur Goldstein blogged recently. “People need reasons to pay, and it’s on leadership to provide them.”

At issue is whether public unions can continue to charge “agency fees,” which are payments collected from people who are not members. Sometimes called a “fair share” fee, it is meant to help unions cover the cost of bargaining contracts that cover all workers, regardless of whether they are union members. Only a fraction of New York City teachers currently opt out of the union and pay the agency fees rather than dues — but experts expect many more teachers could leave the union if the Supreme Court bans the fees.

Mark Janus, a government employee in Illinois, is challenging the fee on the grounds that it violates his right to free speech. The Supreme Court deadlocked on a similar case in 2016 after the sudden death of Justice Antonin Scalia. With Neil Gorsuch now on the bench, observers expect a conservative-leaning court will side with Janus. If that happens, workers covered by unions — including the UFT — will be able to opt out of paying the fees that help keep the unions in operation.

“What that means is there will be a lot of teachers — potentially a lot of teachers in New York — who do not invest in the union,” said Evan Stone, co-founder of the teacher advocacy group Educators for Excellence. “There will be potential growth in free riders who are benefiting from the work of the union without contributing to it.”

That’s why the UFT is kicking into action. The union has trained scores of members to knock on doors and talk to fellow teachers about the case. In about two months, the union estimates its members have knocked on 11,000 doors, sharing stories about how the union has helped them and hoping to convince teachers to keep financially supporting the work, even if the courts decide they’re no longer required to.

Union leaders are also launching “membership teams” in every school. Tasked with “building a sense of unity,” the union is asking the teams to engage in personal conversations with members, and plan shows of support for the union. Stone said his organization is organizing focus groups across the city to inform members about the case.

New York City teachers automatically become union members. They pay about $117 a month in dues, while social workers, paraprofessionals, and members in other school roles pay different amounts. Members can also choose to contribute to a separate political fund, which the union uses to lobby lawmakers and support union-friendly candidates.

About 2,000 educators opt-out of the union and pay agency fees instead — which are the same amount as regular dues, according to a UFT spokesman.

Ken Girardin, who has studied the potential fallout of Janus for New York’s unions as an analyst for the right-leaning Empire Center for Public Policy, said the number of agency-fee payers is low compared to other unions. But the Janus case could change that.

Girardin looked at what happened after Michigan enacted a “right to work” law, which forbid mandatory agency fees. The result: The Michigan Education Association, among the state’s largest unions, saw a 20 percent drop in dues and fees. Among full-time teachers, membership declined by 18 percent.

Girardin estimates an equivalent decrease in New York would mean the state’s teachers unions would take a $49 million hit annually. The UFT relies on dues and agency fees for about 85 percent of its $185 million budget, according to federal documents.

“It means they’d have to make up a course change,” Girardin told Chalkbeat, referring to the potential impact of the Janus decision. “They would have to treat their members like customers instead of people who are going to pay them regardless.”

Behind the scenes, the union is reportedly making contingency plans to deal with the potential budgetary fall-out. The New York Post recently cited unnamed sources who said union leadership is considering reducing the staff at some of its borough offices and cutting back on discretionary spending.

Girardin said public-sector unions in New York have already begun to fight for state legislation that would make it harder for members to drop out — a potential work-around in case the court sides with Janus.

Some UFT members say the threat of Janus is already being felt. The union recently voted down a resolution to support Black Lives Matter after leadership said it was a divisive issue at a time when the union can’t afford to lose members, according to an NY1 report.

Rosie Frascella, a Brooklyn high school teacher who helped organized Black Lives Matter at School events across the city, said she was disappointed in the leadership’s decision. But despite those internal disagreements, she said the threat posed by Janus should compel all teachers to speak out in support of their unions.

“You need to be in a union because it protects your right to teach,” she said. “And it stands up for our students and it creates the schools our children deserve.”

after parkland

As Trump doubles down on call to give teachers guns, the growing #ArmMeWith movement offers an alternative

Counselors, time, diverse classroom libraries, money — these are some of many things American teachers say they need in their schools instead of guns.

The pleas are coming via a social media hashtag, #ArmMeWith, that has spread quickly this week as teachers grapple with the aftermath of last week’s school shooting in Parkland, Florida.

Some lawmakers and advocates — including President Donald Trump — have responded to the shooting by arguing that teachers should be armed. That idea has drawn scorn from educators who argue that more guns in schools would make students less safe and do little to address the underlying issues that contribute to violence in schools.

Now thousands of those educators are offering an alternative, using a template that two teachers shared on Instagram on Tuesday. Olivia Bertels and Brittany Wheaton already had substantial social media followings when they asked others to join them in starting a movement.

“My friend @thesuperheroteacher and I think that we should find more practical solutions than giving teachers guns,” Bertels wrote on her post with the template, where she asked to be armed with school supplies. “I hope you’ll take the same stance.”

More than 5,000 people so far have done exactly that on Instagram, and the hashtag is also trending on Twitter, bringing educators together in a cross-country conversation.

“I wish we didn’t have to do this,” wrote one Texas teacher, HowsonHistory, in a comment on a Rhode Island teacher’s post. “But am so glad that so many teachers are. Maybe soon we will be listened to.”

Here are some of the posts that have caught our eye.

“We, the teachers, have a few ideas.”

“#armmewith not guns, but counselors who do not double as test administrators and more than one overbooked, crowded therapist option for families with Medicaid and social workers without overloaded caseloads.”

“#armmewith the liberation of our students, a microphone to speak out against the policies you make from people who aren’t teachers, resources to empower our children, and love to keep our babies safe. We refuse to be armed with guns. #teachingwhilemuslim”

“Because there are so many other things to be arming ourselves with that will do more good than harm. I choose to #armMeWith kindness not violence and teach my students to do the same #jointhemovement”

“I took my first teaching job the year Sandy Hook happened. And the thing is, in that year and in all the years I have been a teacher since, I have stood in my classroom too many times and wondered where I would put my children if someone came into my classroom with a gun. I have stood on playgrounds and in hallways with dozens of students and wondered what would be the best action to take. I have sat through too many of my lunch breaks with my colleagues hashing over the best strategy for protecting our students. There has to be change. Teachers and students deserve to work and learn in peace. #armmewith #thingsteachersshouldnothavetosay”