human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

Weighing in

Parents rally to demand a voice in the search for New York City schools chief

PHOTO: Courtesy/Shino Tanikawa
Parents and ddvocates rallied on the steps of the New York City education department headquarters to call for a say in the search for a new schools chancellor.

The education department has made it a mission to boost parent involvement in schools. Now, parents are demanding a bigger role elsewhere: In the search for a new schools chancellor.

Parent leaders from across New York City took to the steps of the education department’s headquarters to demand that Mayor Bill de Blasio allow them to have a say in the process.

“For the mayor to deny parents the opportunity to represent the interests of our children in this critical decision is to ignore the voices of our most vulnerable, underrepresented New Yorkers,” Jessamyn Lee, co-chair of the Chancellor’s Parent Advisory Council, said in a statement.

Organizers say about 30 members from a range of parent groups gathered in the rain to call on de Blasio to follow through on a campaign promise made during his first run for mayor.

Before he was was first elected, de Blasio said the city needed a school leader who would be “presented to the public, not just forced down our throat.” But he went on to conduct a hushed search, pulling department veteran Carmen Fariña from retirement to become chancellor.

De Blasio recently won reelection for a second term, and, in December, Fariña announced plans to head back to retirement. This time around, the mayor has committed to a quiet, internal deliberation.

Among the organizations represented at the rally were the Chancellor’s Parent Advisory Council, which is made of leaders from school parent organizations; the Education Council Consortium, which represents members of the local Community and Citywide Education Councils; and the NYCKids PAC, a parent-led political committee. Those are not the only groups seeking more access and transparency in the hiring process. Advocates for different causes, including school integration efforts, have all called for the opportunity to weigh in.

One of those calls came this weekend in an online petition asking de Blasio to consider a well regarded state education official for the job. And the Coalition for Educational Justice, which held its own rally on Tuesday outside City Hall, is calling on the city to appoint a chancellor who “has a strong vision for racial justice in schools.” The organization has called on the city to focus on making sure that teachers have anti-bias training and that classrooms reflect all students’ cultures.

pre-k for all

New York City will add dual language options in pre-K to attract parents and encourage diversity

PHOTO: Christina Veiga
Schools Chancellor Carmen Fariña, back right, visits a Mandarin pre-K dual language program at P.S. 20 Anna Silver on the Lower East Side.

Education Department officials on Wednesday announced the addition of 33 dual language pre-K programs in the 2018-19 school year, more than doubling the bilingual opportunities available for New York City’s youngest learners.

The expansion continues an aggressive push under the current administration, which has added 150 new bilingual programs to date. Popular with parents — there were 2,900 applications for about 600 pre-K dual language seats last year — the programs can also be effective in boosting the performance of students who are learning English as a new language.

Another possible benefit: creating more diverse pre-K classrooms, which research has shown are starkly segregated in New York City.

Schools Chancellor Carmen Fariña said the new programs reflect the city’s commitment to serving all students, even as a national debate rages over immigration reform.

“It’s important to understand that immigrants or people who speak a second language are an asset,” Fariña said. She called bilingual education “a gift that I think all schools should have.”

Included in the expansion are the city’s first dual language pre-K programs in Bengali and Russian, which will open in Jamaica, Queens, and the Upper West Side, Manhattan, respectively. The other additions will build on programs in Spanish, Mandarin and Italian. Every borough is represented in the expansion, with 11 new programs in Manhattan, nine in Brooklyn, six in Queens, five in the Bronx, and two on Staten Island.

In the dual-language model, students split their time between instruction in English and another language. At P.S. 20 Anna Silver, where the recent expansion was announced, pre-K students start the morning in English and transition to Mandarin after nap time. Experts say the model works best when the class includes an equal mix of students who are proficient in each language so they can learn from each other as well as the teacher, though it can often be difficult to strike that balance.

Officials and some advocates view dual-language programs as a tool for integration by drawing middle-class families eager to have their children speak two languages into neighborhood schools that they otherwise may not have considered. Research has shown that New York City’s pre-K classrooms tend to be more segregated than kindergarten. In one in six pre-K classrooms, more than 90 percent of students are from a single racial or ethnic background. That’s compared with one in eight kindergarten classrooms, according to a 2016 report by The Century Foundation.

Sharon Stapel, a mother from Brooklyn, said she knew early on that she wanted her daughter to learn another language and strike relationships across cultures. So she travels to the Lower East Side with her four-year-old, Finch, to attend the Mandarin dual-language pre-K program at P.S. 20 Anna Silver. On Wednesday, the city announced it will add a Spanish dual language program at the school.

“We really see it as how you build community with your neighbors and your friends,” Stapel said. “It was also an opportunity for Finch to become involved and engage in the cultures and in the differences that she could see in the classrooms — and really celebrate that difference.”

Citywide, about 13 percent of students are learning English as a new language. That number does not include pre-K since the state does not have a way to identify students’ language status before kindergarten. However, based on census data, it is estimated that 30 percent of three- and four-year-olds in New York are English learners.

Dual-language programs can benefit students who are still learning English — more so than English-only instruction. Nationally and in New York City, students who are learning English are less likely to pass standardized tests and graduate from high school. In one study, students who enrolled in dual-language courses in kindergarten gained the equivalent of one year of reading instruction by eighth grade, compared with their peers who received English-only instruction.

The city has been under pressure to improve outcomes for English learners. Under the previous administration, New York City was placed on a state “corrective action plan” that required the education department to open 125 new bilingual programs by 2013. Though the city fell short of that goal, the current administration has agreed to place every English learner in a bilingual program by the 2018-19 school year.

Among the greatest barriers to achieving that is finding qualified teachers, Fariña said. In some cases, it can be hard to find teachers who are fluent in the target language. In others, teachers who are native in a foreign language may only be certified in their home country, and it can be hard to transfer that certification to New York.

In order to open an Urdu program recently, Fariña said, the teacher, who holds a degree from another country, went through Teaching Fellows, an alternative certification program that usually caters to career-changers or recent college grads.

“I think the biggest challenge we have right now is ensuring our teacher preparation courses are keeping up with our need and demand for teachers who can teach another language,” she said.