behind the metrics

IBO: City's school progress reports are flawed but an advance

The system the city uses to award letter grades to schools is complicated and in some ways flawed — but it’s the best system we have.

That’s the conclusion of a report by the Independent Budget Office, the city’s budget watchdog that since 2009 has been charged with scrutinizing Department of Education data. The office examined the city’s progress reports, released annually since 2007, to see whether their underlying metrics produce meaningful results.

The progress reports were meant to radically reorient the way that New Yorkers thought about school performance. Instead of assessing schools simply by the proportion of students passing state tests, the progress reports focus on students’ improvement from year to year. In a precursor to the “value-added” measurements now being used to assess teachers, the reports use a complex and evolving algorithm that controls for student demographics to calculate just how much students have progressed.

The city then assigns each school a letter grade based on its score. The letter grades inform both the city’s decisions about which principals should receive bonuses and which schools should be considered for closure and families’ choices are where to enroll.

The IBO concludes that the progress reports offer a more sophisticated analysis of school performance than ever before — but that there is room for improvement. “The methodology used by the education department is a significant improvement over simply basing measures on comparisons of standardized test scores,” the report concludes. “Still, the School Progress Reports have to be interpreted with caution.”

The IBO looked at three issues: whether the city’s algorithm has successfully controlled for factors outside of schools’ control; whether the reports have reflected long-term shifts as well as short-term changes; and whether minor methodology changes produced outsized score swings.

On the first question, the budget office concluded that overall, progress report scores in a small set of schools, those serving both elementary and middle school students, can be considered “demographically neutral” — or unaffected by student characteristics. But in most cases that was not true, according to the IBO’s analysis.

“All other things equal, elementary, middle, and high schools with a higher percentage of black and Hispanic students were consistently likely to have lower overall scores than other schools,” the report notes. Progress report scores were also lower in high schools with more poor students and more students with disabilities, the IBO concluded.

Confirming previous findings, the IBO also concludes that elementary and middle school scores have been highly volatile, with the majority of those schools receiving three or more different progress report grades since 2007. (High school progress report grades are based on a wider range of variables and have always been more stable.) But the IBO says that changes to the reports’ methodology, particularly around how students’ year-to-year growth is assessed, have made them more stable.

Finally, the IBO’s analysis found that most of the changes made to the progress report methodology in 2010 and 2011 did not affect their overall grade. In general, the office concludes, the city’s reports successfully identified very high- and very low-performing schools under multiple methodologies, but they were less successful at distinguishing among middle performers.

“The distinction between a C and D rating for a school may be the result of the particular methodology that the DOE has chosen, among the many that are possible, rather than the result of school practices or effectiveness,” the report concludes. “Unfortunately, this weakness occurs at precisely the point where high stakes decisions about schools are made.”

Department of Education officials say that while the peer group comparisons do not “completely control” for student characteristics, they do reduce the impact of race and other demographics compared to other measures of school performance. They also point out that the IBO found strong correlations between demographics and school grades in only a handful of the relationships it analyzed. And they note that the IBO’s analysis shows that between half and three quarters of C and D grades issued in 2011 would have been the same using the IBO’s methodology for analyzing score stability.

“As the IBO has recognized, New York City’s progress reports are a huge improvement over other state and district systems for measuring student learning in schools — and for this reason they have become a national model,” said spokesman Matthew Mittenthal in a statement. “Closing the achievement gap in New York City is a core goal of our reform strategy, but as long as it exists we should expect it to show up in school progress reports, which are designed to be an accurate reflection of our schools’ strengths and challenges.”

The complete IBO report is below.

defensor escolar

Memphis parent advocacy group trains first Spanish-speaking cohort

PHOTO: Caroline Bauman
Manuela Martinez (center left) and Lidia Sauceda (center right) are among 19 parents in the first Spanish-speaking class of Memphis Lift’s Public Advocate Fellowship.

Manuela Martinez doesn’t want Spanish-speaking families to get lost in the fast-changing education landscape in Memphis as the city’s Hispanic population continues to grow.

The mother of two students is among 19 parents in the first Spanish-speaking class of Memphis Lift’s Public Advocate Fellowship, a program that trains parents on local education issues.

“We want to be more informed,” said Martinez, whose children attend Shelby County Schools. “I didn’t know I had much of voice or could change things at my child’s school. But I’m learning a lot about schools in Memphis, and how I can be a bigger part.”

More than 200 Memphians have gone through the 10-week fellowship program since the parent advocacy group launched two years ago. The vast majority have been African-Americans.

The first Spanish-speaking cohort is completing a five-week program this month and marks a concerted effort to bridge racial barriers, said Sarah Carpenter, the organization’s executive director.

“Our mission is to make the powerless parent powerful …,” she said.

The city’s mostly black public schools have experienced a steady growth in Hispanic students since 1992 when only 286 attended the former Memphis City Schools. In 2015, the consolidated Shelby County Schools had 13,816 Hispanic children and teens, or 12.3 percent of the student population.

Lidia Sauceda came to Memphis from Mexico as a child; now she has two children who attend Shelby County Schools. Through Memphis Lift, she is learning about how to navigate Tennessee’s largest district in behalf of her family.

PHOTO: Caroline Bauman
Hispanic parents attend a training with the Memphis Lift fellowship program.

“Latinos are afraid of talking, of standing up,” Sauceda said. “They’re so afraid they’re not going to be heard because of their legal status. But I will recommend this (fellowship) to parents. How do we want our kids to have a better education if we can’t dedicate time?”

The training includes lessons on local school options, how to speak publicly at a school board meeting, and how to advocate for your children if you believe they are being treated unfairly.

The first fellowship was led by Ian Buchanan, former director of community partnership for the state-run Achievement School District. Now the program is taught in-house, and the Spanish-speaking class is being led this month by Carmelita Hernandez, an alumna.

“No matter what language we speak, we want a high-quality education for our kids just like any other parent,” Hernandez said. “A good education leads to better opportunities.”

Stopping summer slide

On National Summer Learning Day, Memphis takes stock of programs for kids

PHOTO: Helen Carefoot
Torrence Echols, a rising first-grader in Memphis, builds a tower with giant legos at the Benjamin L. Hooks Library on National Summer Learning Day.

When it comes to summer learning, it’s been a better year for Memphis, where a range of new programs have helped to stem learning loss that hits hard in communities with a high number of low-income students.

On Thursday, Mayor Jim Strickland celebrated that work in conjunction with National Summer Learning Day and against the backdrop of the children’s reading room of the city’s main library.

He estimated that 10,000 children and teens are being reached this summer through learning programs spearheaded through Shelby County Schools, Literacy Mid-South, Memphis Public Libraries, churches and nonprofit organizations across the community.

That’s a record-breaking number, Strickland says, in a city with a lot of students struggling to meet state and local reading targets.

Summer learning loss, also known as summer slide, is the tendency for students to lose some of the knowledge and skills they gained during the school year. It’s a large contributor to the achievement gap, since children from low-income families usually don’t get the same summer enrichment opportunities as their more affluent peers. Compounded year after year, the gap widens to the point that, by fifth grade, many students can be up to three years behind in math and reading.

But this summer for the first time, Shelby County Schools offered summer learning academies across the city for students most in need of intervention. And Memphis also received a slice of an $8.5 million state grant to provide summer literacy camps at nine Memphis schools through Tennessee’s Read to be Ready initiative.

Literacy Mid-South used Thursday’s event to encourage Memphians to “drop everything and read!”

The nonprofit, which is providing resources this summer through about 15 organizations in Greater Memphis, is challenging students to log 1,400 minutes of summertime reading, an amount that research shows can mitigate learning loss and even increase test scores.

Reading is a problem for many students in Memphis and across Tennessee. Less than a third of third-graders in Shelby County Schools read on grade level, and the district is working to boost that rate to 90 percent by 2025 under its Destination 2025 plan.

The city of Memphis, which does not fund local schools, has made Memphis Public Libraries the focal point of its education work. This summer, the library is offering programs on everything from STEM and robotics to art and test prep.

Parents are a critical component, helping their kids to take advantage of books, programs and services that counter the doldrums of summer learning.

Soon after the mayor left the Benjamin L. Hooks Library on Thursday, Tammy Echols arrived with her son, Torrence, a rising first-grader at Levi Elementary School. Echols said they visit regularly to read books and do computer and math games.

“We always do a lot of reading and we’re working on learning sight words,” Echols said as she watched her son build a tower out of giant Lego blocks. “Torrence is a learning child and it’s easy to forget what you just learned if you’re not constantly reinforcing.”

You can find summer learning resources for families from the National Summer Learning Association.