human capital

Wide margins of error, instability on city's value-added reports

Some English Language Arts teachers received high "value-added" scores in 2007 but much lower scores in 2008.

The value-added reports meant to measure city teachers’ effectiveness have wide margins of error and give judgments that fluctuate — sometimes wildly — from one year to the next, a new analysis finds.

Schools Chancellor Joel Klein has instructed principals to use the Teacher Data Reports as one way to decide which teachers should receive tenure. Teachers who teach English or math to students in grades three through eight receive the reports.

The NYU economist Sean Corcoran found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

There was an overall correlation between how a teacher scored from one year to the next, and for some teachers, the measurement was more stable. Of the math teachers who ranked in the top quintile in 2007, 40 percent retained that crown in 2008.

The Annenberg Institute for School Reform at Brown University, which has a history of criticizing the Bloomberg administration, published Corcoran’s findings, which were part of a wider look at the practice of assigning “value-added” scores to teachers based on their students’ test scores.

The analysis explains the difference between what value-added scores of teachers aim to do and what value-added measurements actually do in practice. The dream is to isolate the effect of a teacher on students’ performance from the effect of everything else; the reality is that the measures approximate that isolated effect with statistics, weak tests, and small sample sizes.

Corcoran offers some praise. “The simple fact that teachers and principals are receiving regular and timely feedback on their students’ achievement is an accomplishment in and of itself, and it is hard to argue that stimulating conversation around improving student achievement is not a positive thing,” he writes. “But,” he writes,

teachers, policymakers, and school leaders should not be seduced by the elegant simplicity of “value-added.”

The weaknesses of value-added detailed in the report include:

  • the fact that value-added scores are inherently relative, grading teachers on a curve — and thereby rendering the goal of having only high value-added teachers “a technical impossibility,” as Corcoran writes
  • the interference of imperfect state tests, which, when swapped with other assessments, can make a teacher who had looked stellar suddenly look subpar
  • and the challenge of truly eliminating the influence of everything else that happens in a school and a classroom from that “unique contribution” by the teacher

Another challenge for the teachers and principals charged with using value-added scores for self-improvement is the uncertainty about what each individual teacher’s score actually is. On each teacher’s report, the city pinpoints the percentile ranking that represents how she compares to other teachers of the same subject and grade.

But while this is the ranking that the teacher most likely holds, it’s far from 100 percent certain. Indeed, the economists who make value-added scores can only be very certain that the teacher falls somewhere on a range of percentiles (and even getting that cautious, they’re still only 95 percent certain). This range, as you might remember from statistics, is called the “confidence interval.”

For most teachers, the confidence interval is at least 30 percentage points long. For math and English teachers with only one year’s worth of data, the average length is over 60 percentage points. That’s a range of, for instance, between the 10th and 70th percentile of teachers.

The average confidence intervals that Corcoran reports are in the chart below. You can see that, because the confidence intervals shrink as the sample size grows, they are longest when only a year’s worth of data is available.

Teachers in the Bronx face the least certainty. Corcoran guesses that this is because their students are the most likely not to be measured, thereby lowering the data pool — either because the students are classified as special ed or English language learners, and don’t take the state test, or because the students move from year to year, making data about their growth over time harder to come by.

picture-311

The full report is here and below.
The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice

chronically absent

Newark’s absenteeism problem persists as thousands of students miss several days this year, new data show

PHOTO: Patrick Wall/Chalkbeat
Superintendent Roger León addressed ninth-graders in September. That month, about 30 percent of those students were considered "chronically absent."

Thousands of Newark students have already missed multiple school days this year, newly released data show, even as the district’s new superintendent makes improving attendance a top priority.

About one in five students missed more than a week’s worth of class during the first three months of school, according to the district data. Those roughly 8,000 students are already considered “chronically absent.”

Newark has long grappled with exceptionally high rates of chronic absenteeism, which is defined as missing 10 percent or more of school days in an academic year for any reason. Students who miss that much school tend to have lower test scores, higher dropout rates, and greater odds of getting in trouble with the law.

The district’s new superintendent, Roger León, has promised to attack the issue — even going so far as to set a district goal of 100 percent attendance. But the new data, which León released this week, show how far the district has to go.

Nearly 9 percent of of the district’s 36,000 students have already missed the equivalent of more than two school weeks, according to the data. Those 3,200 or so students are labeled “severely chronically absent.”

Experts say that tracking and publicizing attendance data, as León has done, is the first step in combating absenteeism. Now, some district leaders are calling for the next phase of work to begin — analyzing why so many students are missing class and taking steps at the district and school level to help get them to school.

“It’s great that we have all this great data,” Newark Board of Education member Kim Gaddy said at a board meeting last month. “But if you have the data and you’re not using the data to change the situation, we won’t do any justice to our children in this district.”

Students who missed six or more school days from September through November qualify as chronically absent. If they continue at that pace, they are on track to miss the equivalent of a month or more of school by June. Students who missed 10.5 days or more during those three months count as severely chronically absent.

Attendance from Sept. to Nov. 2018. | Green = absent 0-2.5 days | Yellow = absent 3-5.5 days | Orange = absent 6-10 days | Red = absent 10.5 or more days. | Credit: Newark Public Schools

The early data show that Newark’s long-standing absenteeism patterns are continuing. The chronic absenteeism rates over the past three months were about the same as in 2016, according to the data.

The problem remains most acute among the district’s youngest and oldest students: 41 percent of pre-kindergarteners were chronically absent this November, as were 45 percent of 12th-graders. At least a third of students at five high schools — Barringer, Central, Malcolm X Shabazz, Weequahic, and West Side — were severely chronically absent last month.

While absenteeism rates varied among schools, they tended to be highest in the city’s impoverished South Ward.

“I’m concerned particularly about the South Ward,” Gaddy said at the Nov. 20 board meeting. “That’s where our children need the most assistance.”

León, a former principal who became schools chief on July 1, has already taken some early steps to improve attendance.

His most visible effort was a back-to-school campaign called “Give Me Five,” where he ordered every district employee to call five families before the first day of school. The campaign, for which León himself recorded robocalls to families, appears to have made a difference: 91 percent of students showed up the first day, the highest rate of the past four years, according to district data. (In 2013, when former Superintendent Cami Anderson launched her own attendance campaign, about 94 percent of students attended school the first day.)

The district also eliminated some early-dismissal days, which typically have low attendance. And students with mid-level test scores whom León has targeted for extra support have had better attendance this year than their peers, officials said.

Attendance in Nov. 2018 | Green = 0-0.5 days absent | Yellow = 1-1.5 days absent | Orange = 2-2.5 days absent | Red = 3 or more days absent | Credit: Newark Public Schools

However, León hit a snag trying to enact the crux of his attendance plan — reinstating more than 40 attendance counselors whom Anderson laid off years ago to cut costs. The state’s civil service commission has said the district must offer the jobs to the laid-off counselors before hiring anyone new, León told the board — forcing the district to track down former employees who, in some cases, have moved to different states. Only eight counselors have been hired to date, but León said he hopes to fill the remaining positions next month.

Meanwhile, León is arguing that some of the responsibility for improving attendance falls on families. At November’s board meeting, he said some parents and guardians “believe that, in fact, they can keep their children home” from school. At a parent conference this month, he took that message directly to families.

“I don’t care if school ends at 10 and they’re only going to come for an hour, and half an hour is on a bus,” he told several hundred parents who showed up for the daylong summit. “When I tell you that your child is coming to school, it’s your job to make sure the child comes to school.”

Afterwards, several parents and school employees said they welcomed León’s tough talk on attendance.

“It was about parents ensuring kids are in school and they are doing good,” said Bilikis Oseni, who has a child in first grade at Camden Street School. “Attendance is very key.”

Still, Newark families face many obstacles in getting their children to school, according to a 2016 report on chronic absenteeism among young students. Parents cited a lack of school busing, asthma and other childhood health problems, and work schedules that make it hard to drop off their children in the morning. High school students listed uninspiring classes, mental-health challenges, and safety concerns when traveling to school as reasons why they don’t show up, according to a 2017 report.

At the November board meeting, several members asked León whether he planned to dig deeper into the causes of absenteeism.

“I was looking through all the statistics here in the packet,” said Andre Ferreira, the board’s student representative, who attends Science Park High School. “But there were none that looked towards having a survey of students themselves telling you why they aren’t coming to school.”

León noted that he held forums with high-school students in September where he stressed the importance of showing up. He also said he has a student-only email address that some students have used to explain why they miss school.

“So I’m gathering data that lets me know why a particular student in fact hasn’t been to school,” he said. “Ultimately, we would have to do that for every single student, in every classroom, in every grade, in every school. That’s really the work — and it’s hard to do.”

Peter Chen, who co-authored the two reports on chronic absenteeism in Newark, said the superintendent had taken a crucial first step by raising awareness about the city’s attendance challenges. The district also appears to be sharing attendance data more regularly with schools, he said.

The next step is for the district to help schools identify and assist students who are chronically absent. The central office can do that by sharing effective attendance strategies, training school workers on how to support students’ social and emotional well-being, and offering grants to fund schools’ own attendance campaigns, he added.

“This is something that requires tailored, school-level responses,” said Chen, who is a policy counsel for Advocates for Children of New Jersey. “The district can help support some of that — but it’s not something that’s easy to impose from on high.”

Newark Enrolls

Want to attend one of Newark’s coveted magnet schools? Get ready to take a test.

Newark students who hope to attend one of the district's six magnet schools will have to take a new exam in January.

Newark students will soon face a new hurdle when trying to snag seats at the city’s most popular high schools.

Next month, any student who wants into one of the city’s six magnet schools will have to take a new exam that gauges their academic prowess as well as their interest in each school’s theme.

“If you would like to go to any of those schools,” Superintendent Roger León told parents at a conference Wednesday, “you better get ready for the test to get in.”

The exam, which will be given to students on Jan. 11-12, has not yet been announced on the district’s enrollment site. In fact, the test itself is still being developed and logistical details, such as where students will take it, are still being determined, officials said.

In addition to the new test, each school will also begin interviewing applicants, León said — something only two magnet schools did last year, according to an admissions guide. It’s unclear whether the interviews will take place this admission cycle. If so, schools may have to schedule dozens or even hundreds of interviews in the coming weeks.

Meanwhile, enrollment for next school year began on Dec. 3 and continues through February — giving students and schools little time to prepare for the new requirements.

“I know as much as you know right now,” one principal said. “Obviously the superintendent is revamping some items, but he hasn’t really shared the details with everyone.”

The district-run magnet schools, which have themes such as science and American history, include some of the city’s most sought-after high schools. Last year, nearly 1,800 eighth-graders listed a magnet school first on their high-school applications even though the schools had space for only 971 students.

The magnets, which vastly outperform the district’s six traditional high schools, already screen applicants. They look at grades, state test scores, attendance records, and — in the case of Arts High School — an audition or visual-art portfolio, when deciding which students to let in.

But even with those screens, some admitted students are not prepared for the rigor of work at the magnets or lack a strong interest in their programs, León said.

“The idea is to make sure that students who choose to go to these schools are going to meet whatever are the demands of that school,” he told Chalkbeat. “It’s not that your parents have the right to choose for you to go.”

Even as he moves to make magnet schools more selective, León — who became superintendent in July — also hopes to make traditional schools more appealing to top students.

On Wednesday, he also announced plans to create gifted-and-talented programs at each of the traditional high schools. To qualify, students will also take the new magnet-school exam.

León did not go into detail about what the programs will entail. But he may be drawing from his previous experience as principal of University High School, a magnet school that advertises a gifted-and-talented program on its website. Students must test into the program, which includes a “rigorous curriculum” in English, math, and another language, according to the site.

“Students are going into magnets because they think that’s where they can get their high-performing education,” he said. “Now they’ll be given a reason to not do that.”

The traditional schools will also develop specialized “academies” to train students for various careers, including engineering, teaching, and health services. Each school will partner with a higher-education institution and a professional organization to develop those programs.

Many Newark schools have tried to offer vocational programs, but often struggle to find qualified teachers and meet the stringent requirements to receive federal funding. It’s unclear how the district will help them overcome those challenges, especially if the timeline is also aggressive.

Traditional schools, for their part, seem eager for any support they can get. Angela Mincy, principal of Barringer High School, said the school created an honors program last year in an effort to retain high-achieving students.

“If I don’t create an isolated experience for them, I will lose them,” she said in an interview last month, adding that the goal is to keep attracting more and more top students. “The hope is that one day, one honors track will become two will become three.”

With their selective admissions and college-oriented courses, the city’s magnet schools have long been seen as a refuge for high-achieving students who cannot afford private school. County-run vocational schools, which also screen applicants, are another popular option along with some charter high schools — though they often have few seats left over for students who did not attend their lower-grade schools.

The district’s traditional, or “comprehensive,” high schools are viewed by many families as schools of last resort. On nearly every academic measure — attendance rates, test scores, college enrollment and completion — the traditional schools lag far behind the magnets.

In a sense, this disparity is built into the system. Magnet schools are designed to enroll academically and artistically accomplished students. Traditional schools take the rest, including almost all students who are still learning English and the majority of those with disabilities.

Other cities have begun to rethink this practice of tracking students into separate schools according to ability — at least as measured by a single test. In New York City, where a debate has raged over admissions to the district’s coveted “specialized” high schools, the mayor has proposed scrapping the schools’ entrance exam. Instead, he said, they should reserve spots for the top students from every middle school.

Some Newark parents have floated a similar plan for the city’s most exclusive magnet school, Science Park High School. Instead, Superintendent León is pursuing the opposite approach — adding new entrance exams for all magnet schools. In other cities, exam schools tend to be highly segregated by race and class, favoring families with the wherewithal to help students prepare for the exams or pay for test prep.

León said he expects the new magnet exams will measure students’ reading and math proficiency, as well as their interest in each school’s particular focus, such as science or technology.

“The whole concept that anyone and everyone can get into the magnet high schools — that’s not why they were designed,” said León, who graduated from Science High School. “You actually have to qualify to get into those schools.”

Karen Gaylord, Science Park High School’s community engagement specialist, said some teachers and  parents may grumble about the new test because they haven’t had a chance to prepare students for it. But she noted that families had become “resigned” to entrance exams when magnet schools used them in the past.

She also said many people would welcome the admissions interviews as a way for students to highlight skills and interests that aren’t reflected on their transcripts. The question, she said, is how schools will carry out these changes on such a tight timeline.

“It feels like there are so many opportunities to get this right,” she said. “I’m just not sure we’re going to get them all in this year. The clock is ticking.”