human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Betsy DeVos

To promote virtual schools, Betsy DeVos cites a graduate who’s far from the norm

U.S. Education Secretary Betsy DeVos spoke to the National Alliance for Public Charter Schools in June.

If Betsy Devos is paying any attention to unfolding critiques of virtual charter schools, she didn’t let it show last week when she spoke to free-market policy advocates in Bellevue, Washington.

Just days after Politico published a scathing story about virtual charters’ track record in Pennsylvania, DeVos, the U.S. education secretary, was touting their successes at the Washington Policy Center’s annual dinner.

DeVos’s speech was largely identical in its main points to one she gave at Harvard University last month. But she customized the stories of students who struggled in traditional schools with local examples, and in doing so provided an especially clear example of why she believes in virtual schools.

From the speech:

I also think of Sandeep Thomas. Sandeep grew up impoverished in Bangalore, India and experienced terrible trauma in his youth. He was adopted by a loving couple from New Jersey, but continued to suffer from the unspeakable horrors he witnessed in his early years. He was not able to focus in school, and it took him hours to complete even the simplest assignment.

This changed when his family moved to Washington, where Sandeep was able to enroll in a virtual public school. This option gave him the flexibility to learn in the quiet of his own home and pursue his learning at a pace that was right for him. He ended up graduating high school with a 3.7 GPA, along with having earned well over a year of college credit. Today, he’s working in finance and he is a vocal advocate for expanding options that allow students like him a chance to succeed.

But Thomas — who spoke at a conference of a group DeVos used to chair, Advocates for Children, in 2013 as part of ongoing work lobbying for virtual charters — is hardly representative of online school students.

In Pennsylvania, Politico reported last week, 30,000 students are enrolled in virtual charters with an average 48 percent graduation rate. In Indiana, an online charter school that had gotten a stunning six straight F grades from the state — one of just three schools in that positionis closing. And an Education Week investigation into Colorado’s largest virtual charter school found that not even a quarter of the 4,000 students even log on to do work every day.

The fact that in many states with online charters, large numbers of often needy students have enrolled without advancing has not held DeVos back from supporting the model. (A 2015 study found that students who enrolled in virtual charters in Michigan, Illinois, and Wisconsin did just as well as similar students who stayed in brick-and-mortar schools.) In fact, she appeared to ignore their track records during the confirmation process in January, citing graduation rates provided by a leading charter operator that were far higher — nearly 40 points in one case — than the rates recorded by the schools’ states.

She has long backed the schools, and her former organization has close ties to major virtual school operators, including K12, the one that generated the inflated graduation numbers. In her first week as education secretary, DeVos said, “I expect there will be more virtual schools.”

Correction: An earlier version of this article misstated the location of the dinner.

expansion plans

Here are the next districts where New York City will start offering preschool for 3-year-olds

PHOTO: Christina Veiga
Schools Chancellor Carmen Fariña, left, and Mayor Bill de Blasio, center, visited a "Mommy and Me" class in District 27 in Queens, where the city is set to expand 3-K For All.

New York City officials on Tuesday announced which school districts are next in line for free pre-K for 3-year-olds, identifying East Harlem and the eastern neighborhoods of Queens for expansion of the program.

Building on its popular universal pre-K program for 4-year-olds, the city this year began serving even younger students with “3-K For All” in two high-needs school districts. Mayor Bill de Blasio has said he wants to make 3-K available to every family who wants it by 2021.

“Our education system all over the country had it backwards for too long,” de Blasio said at a press conference. “We are recognizing we have to reach kids younger and more deeply if we’re going to be able to give them the foundation they need.”

But making preschool available to all of the city’s 3-year-olds will require an infusion of $700 million from the state or federal governments. In the meantime, de Blasio said the city can afford to expand to eight districts, at a cost of $180 million of city money a year.

Funding isn’t the only obstacle the city faces to make 3-K available universally. De Blasio warned that finding the room for an estimated 60,000 students will be a challenge. Space constraints were a major factor in picking the next districts for expansion, he said.

“I have to tell you, this will take a lot of work,” he said, calling it “even harder” than the breakneck rollout of pre-K for all 4-year-olds. “We’re building something brand new.”

De Blasio, a Democrat who is running for re-election in November, has made expansion of early childhood education a cornerstone of his administration. The city kicked off its efforts this September in District 7 in the South Bronx, and District 23 in Brownsville, Brooklyn. More than 2,000 families applied for those seats, and 84 percent of those living in the pilot districts got an offer for enrollment, according to city figures.

According to the timeline released Thursday, the rollout will continue next school year in District 4 in Manhattan, which includes East Harlem; and District 27 in Queens, which includes Broad Channel, Howard Beach, Ozone Park and Rockaways.

By the 2019 – 2020 school year, the city plans to launch 3-K in the Bronx’s District 9, which includes the Grand Concourse, Highbridge and Morrisania neighborhoods; and District 31, which spans all of Staten Island.

The 2020 – 2021 school year would see the addition of District 19 in Brooklyn, which includes East New York; and District 29 in Queens, which includes Cambria Heights, Hollis, Laurelton, Queens Village, Springfield Gardens and St. Albans.

With all those districts up and running, the city expects to serve 15,000 students.

Admission to the city’s pre-K programs is determined by lottery. Families don’t have to live in the district where 3-K is being offered to apply for a seat, though preference will be given to students who do. With every expansion, the city expects it will take two years for each district to have enough seats for every district family who wants one.