human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

To and Through

Newark’s post-grad paradox: More students are entering college, but few earn degrees

PHOTO: Patrick Wall/Chalkbeat
Newark Mayor Ras Baraka wants 25 percent of residents to have college degrees by 2025, up from 19 percent today.

When it comes to college, Newark faces a good news-bad news paradox.

More students than ever are graduating high school and enrolling in college, according to a new report. Yet fewer than one in four Newark students earns a college degree within six years of graduating high school — leaving many with limited job prospects in a city where an estimated one-third of jobs require a four-year college degree.

Now, city officials are promising to build on the report. They want to ramp up the rigor of high-school classes and create more early-college programs to increase the odds of students entering college and leaving with a degree.  

“How do we teach our children to perform — to graduate?” Mayor Ras Baraka asked at a press conference Wednesday to mark the official release of the report of Newark students’ college outcomes. “We got them in the door,” he said of students who attend college. “Now how do we make them stay?”

The city’s plans, to which Superintendent Roger León is lending his support, reflect a growing recognition that simply getting students into college is not sufficient — and can even backfire if they drop out before graduation, leaving them with college debt but no degree.

Until recently, the charge given to high schools in Newark and across the country was to foster “college-going cultures.” And these efforts showed promising results: On average, 51 percent of Newark Public School students who graduated high school between 2011 and 2016 immediately enrolled in college, up from 39 percent who did so between 2004 and 2010, according to the report by the Newark City of Learning Collaborative, or NCLC, and Rutgers University-Newark’s School of Public Affairs and Administration.

But entering college didn’t guarantee its completion. Of those students who started college straight after high school, only 39 percent earned a degree within six years, the report found.

As a result, educators and policymakers have begun to think harder about how to help students “to and through” college — to ensure they actually earn degrees. Toward that end, Baraka and the NCLC — which includes roughly 40 colleges, schools, nonprofits, and corporations — has set a goal of 25 percent of Newark residents earning college degrees or comparable credentials by 2025.

Today, just 19 percent of Newark adults have associate degrees or higher — compared to 45 percent of adults across New Jersey and 40 percent nationally.

Superintendent León, who began overseeing the city’s schools on July 1, said his main strategy for supporting these efforts will be to expose students to challenging work early on.

“If we don’t do something dramatically in classrooms to improve instruction and make it rigorous,” León said after Wednesday’s event, then students are “getting into college but they’re not completing it.”

Source: “Post-Secondary Outcomes of Newark High School Graduates (2011-2016)” report. Note: The four-year rate is an average of the classes of 2011 to 2013. The six-year rate is from the class of 2011. Graphic: Sam Park/Chalkbeat

For starters, León said he wants high schools to offer more college-level classes. In the 2016-17 school year, just 21 percent of Newark students were enrolled in one or more Advanced Placement or International Baccalaureate classes — compared to 42 percent of students statewide.

He also vowed to raise the quality of instruction in the district’s traditional high schools. Only 14 percent of their graduates earn college degrees within six years, compared to 42 percent of graduates from the city’s selective magnet schools, the report found.

To do that, León said he will create specialized academies within the traditional schools modeled on the magnets, which have specialized themes such as science, technology, or the arts. The academies, which will partner with colleges, will most likely feature admissions criteria similar to those of magnet schools, which select students based on their academic and attendance records, León added.

And, for the first time, all ninth-grade students this academic year will take the Preliminary SAT, or PSAT, León said Wednesday. An additional 1,100 eighth-graders who passed at least one of their seventh-grade PARCC exams will also take the PSAT when it’s administered on Oct. 10.

Since 2016, the district has provided the PSAT to all 10th and 11th-grade students. But León said that giving the test to younger students will focus their attention on college and help identity those who are ready for advanced classes. The PSAT is designed to help students prepare for the SAT, which is used in college admissions, and to qualify for National Merit Scholarships.

The district, which was under state control for 22 years until February, is getting some assistance in its effort to improve students’ college outcomes.

For instance, KIPP, the national charter-school network with eight schools in Newark, is sharing its strategies for helping students choose the right college with guidance counselors at three district high schools.

And the higher-education institutions in the Newark City of Learning Collaborative, including Essex County College and Rutgers University-Newark, plan to create more “dual-enrollment” programs that allow high-school students to earn college credits, said NCLC Executive Director Reginald Lewis.

“We’re all going to do a better job,” Lewis said, “of making sure that once Newark residents get in our doors, that we help them persist.”

Time crunch

In victory for teachers union, Newark superintendent scraps longer hours for low-performing schools

PHOTO: Patrick Wall/Chalkbeat
Superintendent Roger León at Hawkins Street School, one of the schools that will lose its extended hours.

Newark’s new superintendent is eliminating a program that extended the hours of struggling schools, which the teachers union has long attacked as ineffective and unfair to educators.

Teachers at roughly 30 schools will no longer receive $3,000 annual stipends for the extra hours, a provision written into the current teachers contract, which extends to 2019. Instead, all 64 district schools will get extra funding for before and after-school programs, Superintendent Roger León said in an email to employees on Tuesday.

The changes will go into effect Monday, Sept. 10, resulting in new hours for the affected schools just days after the new school year began. The district is still working to adjust pickup times for students who are bused to school, according to León’s email. A few of the schools will phase out their extended hours later in the year, the email said.

“We will not continue to do the same things as before and be surprised when the results do not change,” León wrote, adding that cutting the extra hours would save the district $5 million.

In an interview with Chalkbeat Thursday, León said the move is intended to create more uniformity among schools and the services they provide. Now, all schools will get additional money to pay for programs outside of the regular school day, which schools can tailor to their individual needs, though students who are struggling academically will continue to receive “intensive” support, he said.

“Ultimately, the idea would be by October having completely different after-school and before-school programming that meets the needs of each respective school,” León said.

The extended time was first included in the teachers contract in 2012 as part of a larger improvement plan for the targeted schools, which was developed by Cami Anderson, Newark’s former state-appointed superintendent. The plan also designated some low-performing schools as “renew” schools, where teachers had to reapply for their positions and work longer hours.

Anderson also closed some schools and gave principals new hiring authority. Both actions left dozens of tenured teachers without positions, so Anderson created a fund to pay those teachers to perform support duties in schools. In 2014, that fund for “employees without placement” cost the district $35 million out of its nearly $1 billion budget, though by last year the fund had shrunk to $8 million for about 100 unassigned teachers, according to officials.

León said in Tuesday’s email that he was also eliminating the fund, which he said would save the district another $6 million. The teachers union president said he believed all the unassigned teachers now have placements, but the district did not respond to a request to confirm that.

León is also removing the “renew” and “turnaround” labels from low-performing schools, citing their “progress and student achievement,” according to the email.

“I applaud everyone’s efforts at renew or turnaround schools and acknowledge what has been accomplished,” he wrote.

Now that León has abolished his predecessors’ school-improvement program, he will be expected to create his own. Many schools remain mired in poor performance, even as the district overall has made strides in recent years.

When the teachers union agreed to the extended hours in its 2012 contract with the district, it was hailed nationally as a major breakthrough in efforts to revamp troubled schools. But even as the union agreed last year to keep the provision in its current contract, union officials have assailed the turnaround effort as a failure.

NTU President John Abeigon told Chalkbeat on Thursday that the program had been a “scam” and “nothing more than extended childcare.” He added that the stipend teachers received amounted to about $7 per hour for the extra time they worked.

In 2016, a district-commissioned survey of 787 teachers at schools with extended hours found that two-thirds of teachers at schools where the extra time was spent on student instruction said the time was valuable. But in a survey the union conducted in April, the 278 teachers who responded gave the extended hours low ratings for effectiveness in boosting student achievement.

Some teachers in the union survey praised the longer hours, saying their schools used them effectively to lengthen class periods, run after-school clubs, or allow teachers to plan lessons or review student data. But others said the extra time was squandered, leaving staff and students exhausted with little evidence of improved student outcomes to show for it. (Students’ pass rates on state tests stayed flat or declined at most “renew” schools in the first years of the program.)

The union also has complained that many teachers felt compelled to work the extra hours because those who refused to could be transferred to different schools. Under the terms of the original extended-day agreement, teachers were required to work an extra hour per day and attend trainings during the summer and some weekends.

In León’s email to employees, he said every extended-day school had set different work requirements and “none are consistent with the original design.” The longer days may also be contributing to high teacher turnover in those schools, he wrote, adding that principals of schools with regular hours told him they did not want to extend their hours.

Abeigon, the union president, applauded León’s decision to scrap the extra work hours.

“He came to the conclusion that we expected any true educator to reach: that the program was not working and was never going to work,” he said.

León said Thursday that he is now working on a new turnaround program. Once it’s ready, he promised to share the details with affected families before publicly announcing which schools are part of it — an effort to avoid the student protests that erupted when Anderson identified her “turnaround” schools.

He also said he was still considering whether he would ever close schools that fail to improve or to reverse their declining enrollments. Anderson’s decision to shutter nearly a dozen long-struggling schools continues to fuel resentment among her critics even years later.

“I think the whole idea of how much time does a school get to correct itself is a very important one and I’m going to need to be really reflective on it,” León said. “I’ve seen what closing schools does with people who do not feel that they were aware of it or a part of fixing it.”