First Person

Survivorship Bias And The Hidden Costs Of Backfill

Out of 90 charter schools that administered the New York State standardized tests in both 2011 and 2012, Harlem Link had the eighth-highest average increase in English language arts and math scores. This score improvement was amazing, fantastic, even inspiring. And misleading — because of a small, relatively unknown factor called “survivorship bias.”

Survivorship bias is a statistical term for an indication that there is some hidden factor that excludes certain members of a data set over time — namely, part of a sample that was there at the beginning is no longer there at the end and does not count in the final analysis. The smaller subset of those who “survive” over time might be better off than the original whole group simply because of who stayed and who left, not any value added over time.

Simply put, every year, at every school, some students leave, and their departure changes the profile of who takes the test from year to year. Sometimes high-scoring students depart. At other times, low-scoring students depart.

If schools continuously enroll new students (and some don’t), the same factor impacts the student population for these incoming students. At the end of this piece I chart a hypothetical situation in which survivorship bias shows how a school can appear to improve while not actually adding any value simply by not adding new students year after year.

In large systems, there is so much mobility that these student profiles tend to cancel each other out because of scale. For example, the student population appears relatively stable from year to year in the third grade in Community School District 3, where 1,342 students in 30 schools took the state English Language Arts exam in 2012. But in small student populations like the one at Harlem Link, where only 52 third-grade students took the 2012 exam, a few students entering or leaving the school with certain test scores can make a big difference.

When the state department of education releases test scores each year, however, it does not provide this or any other contextual background information alongside the scores. I believe that this process penalizes, in the public eye, schools that continue to enroll students to replace those that depart.

(Partly) illusory gains

At Harlem Link, the fact that we only test in three grades guarantees that at least one third of our students taking the tests each year will be different students from those who took it the year before. Putting aside the variability in the state test from year to year, this rolling of the dice has influenced some dramatic swings in achievement that mean our school’s test scores have looked worse than the actual performance of our teachers in some years, and at other times (like this year) they may have looked better than they really were.

It turns out that the profile of our students who departed before the last school year was a much less successful one than the profile of the group that left the prior year. In other words, we had to improve less to get apparently lofty gains.

In English Language Arts, we saw an improvement of 18 percentage points from 2011 to 2012, according to the state’s way of reporting the scores. But since many of the students who graduated in 2011 or left for other reasons following the 2010-2011 academic year performed poorly on the 2011 exams, the students who returned had a better passing rate by 10 percentage points than the original group. In other words, more than half of our test score gains in ELA could be accounted for by attrition.

Now, I’m not going to say that I’m not proud of our scores or that they are not indicative of a powerful effort by talented and dedicated professionals. I’m not even going to tell you that we didn’t improve our practice last year. I think we have improved in that area every year, because we have been an honest, self-examining, learning organization. But the wide swing in test scores and the state’s failure to describe enrollment patterns when reporting the scores masks the true story of a gradual, continuous march to improvement that is the real hallmark of the growth at Harlem Link.

Best practices often begin as difficult, controversial and seemingly impossible changes to “the way things are.” Strong schools take the time required to plan, assess, and tweak new initiatives until they become standard operating procedures. The lack of information provided alongside scores obscures this type of growth, creating perverse incentives for schools to “push out” students who are low performers and to “quick fix” by whittling down large original cohorts to smaller groups of survivors, uncompromised by new admittees.

At Harlem Link, we have resisted these perverse incentives. We have always replaced students who leave, for budgetary reasons (being a small, standalone charter school) and to serve a greater portion of the community starved for high-quality school choices. Each year, we have encouraged some students who are particularly high-achieving to leave a year early by helping them apply to competitive public and independent middle schools that only admit in fifth grade, reasoning that we’d rather lose their strong fifth-grade test scores than see them lose an opportunity to get firmly on the college track a year ahead of their peers.

If we followed the short-sighted state incentive, we would not have urged four of our highest-scoring fourth-graders on the state exams in 2012 to apply to and enter the Upper West Side’s highly sought-after Center School. They were admitted and are all attending — a fact that may well push down our fifth-grade test scores by as much as 10 percent next year — and we are thrilled, because we helped four more students living in a high-poverty environment to gain admission to this exclusive public school.  We also would not have pushed students leave after fourth grade in years past to embark on the independent school track by attending the East Harlem School at Exodus House and the George Jackson Academy in lower Manhattan.

In the context of reform

This issue has been raised before in the blogosphere, but not in a thoughtful manner. Instead, it has been wielded as a weapon by those who are against the current strain of education reform. It has been used to defeat the straw-man argument that charters are silver bullets and to denigrate the success of networks like KIPP, which is another organization that deserves no such uninformed criticism. (Each year, KIPP asks itself several questions in its annual internal reporting, including, “Are we serving the students who need us?”)

Because it is potentially embarrassing and might burst the balloon of so-called charter education miracles, this issue has also (to my knowledge) been ignored publicly by my colleagues in the charter community. There are many groups of charter schools that go happily on their way winnowing down their large kindergarten classes, educating fewer and fewer students in each cohort each year, not adding new students and narrowing down their challenges as they deal with fewer and fewer “survivor” students well. And those charters that benefit from network infrastructure and economies of scale can balance their budgets even while shrinking six to eight kindergarten sections down to three or four fifth-grade sections.

I’m not passing judgment on those networks. As a charter school founder who has been running a school for almost 10 years, I still believe that the charter experiment has been a profoundly positive one for the communities where such schools have flourished. What I want is for the public to have some understanding of the context behind test scores, so alleged miracles can be put in their proper place, and year-to-year statistical swings that have nothing to do with a school community’s actual performance can be put into their proper perspective.

Hypothetical (with some assumptions): Survivorship bias in action

In the example below, compare two schools that start out with similar student profiles. School A replaces each student who departs. School B does not.

Each year at both schools, a greater percentage of academically struggling students than successful students leave. Each year at both schools, neither school is adding any value since no individual’s test scores are changing.

Because the entering students at School A are similarly academically disadvantaged to those who depart, its scores do not change. School B’s scores improve more than 20 percentage points — simply by virtue of attrition, the decision not to enroll new students, and the mix of which students are taking the test each year.

School A enrolls new students continuously

School B does not enroll new students

Steve Evangelista is the principal of Harlem Link Charter School. This piece originally appeared on the school’s blog.

First Person

I covered Tennessee’s ed beat for Chalkbeat. Here’s what I learned.

PHOTO: Marta W. Aldrich
Grace Tatter covers a press conference at the Tennessee State Capitol in 2015.

For three years, I covered the Statehouse for Chalkbeat Tennessee, reporting on how policies from Nashville trickled down into more than 1,800 public schools across the state.

Now I’m starting back to school myself, pursuing graduate studies aimed at helping me to become a better education journalist. I’m taking with me six things I learned on the job about public education in Tennessee.

1. Apathy is often cited as a major problem facing education. That’s not the case in Tennessee.

I heard from hundreds of parents, educators, and students who were passionate about what’s happening — good and bad — inside of schools. I covered crowded school board meetings and regularly scrambled for an open seat at legislative hearings where parents had filled the room after driving since dawn to beat the opening gavel. Not incidentally, those parents usually came from communities with the “worst” schools and the lowest test scores. While many disagreements exist about the best way to run schools, there is no shortage of people, particularly parents and educators, who care.

2. Tennessee has one of the most fascinating education stories in America.

I’ve had a front-row seat to massive changes in K-12 education under reforms ushered in by Race to the Top — an overhaul being tracked closely well beyond the state’s borders. But the national interest and import doesn’t end with changes stemming from the $500 million federal award. Tennessee is home to some of the nation’s premier education researchers, making its classrooms laboratories for new ideas about pre-K, school turnaround, and literacy instruction, just to name a few. And at the legislature, more lobbyists are devoted to education than to most any other cause. A lot of eyes are on Tennessee schools.

3. The education community is not as divided as it looks.

During the course of just a few years, I watched state lawmakers change their positions on accountability and school vouchers. I witnessed “anti-charter” activists praise charter leaders for their work. I chronicled task force meetings where state leaders who were committed to standardized testing found middle ground with classroom educators concerned that it’s gone too far. In short, a lot of people listened to each other and changed their minds. Watching such consensus-building reminded me that, while there are no simple debates about education, there is a widespread commitment to making it better.

4. Money matters.

Even when stories don’t seem to be about money, they usually are. How much money is being spent on testing, teacher salaries, school discipline reform? How much should be available for wraparound services? Why do some schools have more money than others? Is there enough to go around? Tennessee leaders have steadily upped public education spending, but the state still invests less than most other states, and the disparities among districts are gaping. That’s why more than a handful of school districts are battling with the state in court. Conversations about money are inextricable from conversations about improving schools.

5. Race is a significant education issue, but few leaders are willing to have that conversation.

More than 60 years after Brown v. Board of Education, Tennessee’s schools are largely racially segregated. Yet most policymakers tread lightly, if ever, into conversations about achieving real racial integration. And in many cases — such as a 2011 law enabling mostly white suburban Shelby County towns to secede from the mostly black Memphis district — they’ve actually gone backwards. Then there’s the achievement data. The annual release of test scores unleashes a flurry of conversation around the racial achievement gap. But the other 11 months of the year, I heard little about whether state and local policies are closing those gaps — or contributing to them — or the historical reasons why the gaps exist in the first place. To be sure, state leadership is trying to address some of Tennessee’s shortcomings. For example, the State Department of Education has launched modestly funded initiatives to recruit more teachers of color. But often, race and racism are the elephants in the room.

6. Still, there’s lots to celebrate.

If there were unlimited hours in the day, I could have written thousands of stories about what’s going right in public education. Every day, I received story ideas about collaborations with NASA in Oak Ridge, high school trips to Europe from Memphis, gourmet school lunches in Tullahoma, and learning partnerships with the Nashville Zoo. Even in schools with the steepest challenges, they were stories that inspire happiness and hope. They certainly inspired me.

Grace Tatter graduated from public schools in Winston-Salem, N.C., and received her bachelor’s degree in history from the University of North Carolina. She’s now pursuing a master’s degree in specialized studies at the Harvard Graduate School of Education.

First Person

I’m a Houston geography teacher. This is my plan for our first day back — as soon as it arrives

PHOTO: Creative Commons / Texas Military Department
Texas National Guard soldiers arrive in Houston, Texas to aid citizens in heavily flooded areas from the storms of Hurricane Harvey.

Hurricane Harvey has upended so many things here in Houston, where I am starting my third year as a teacher. One of them is the lesson I am planning for the first day of school — as soon as it arrives.

This upheaval is nothing compared to what people across the city have faced, including my students, who have been sending me photos of evacuation boats going past their houses.

But it is fundamental to the task of being a teacher at a time of crisis. As an A.P. Human Geography teacher, my job is to help students make connections between the geography concepts we are learning in class and their real lives: Does Houston look like the models of urban development we study? Does their family history include a migration?

Before the storm, my thinking went like this: I am white and was born in England and most of my students are Hispanic, many with parents who were born in other countries. I was excited for us to share and compare our different stories. My students last year were shocked and fascinated when they discovered that my white, middle-aged father who is a university professor was applying for a green card, just as many of their family members were.

Now, Hurricane Harvey has underlined for me the importance of those real-world connections. As I looked at the photos from my students, I was struck by how geography concepts can affect us in very real — even life-threatening — ways.

I had planned to teach a lesson at the end of the year about how urbanization affects the environment. The lesson looks at how urbanization can exacerbate flooding: for example, how paving over grassy areas can increase the speed with which rain reaches the bayous, causing the water levels to rise faster. I would then have students evaluate different policies cities can adopt to mitigate that risk, such as encouraging the building on brownfield rather than greenfield sites and passing laws to protect farmland — options that have significant benefits but also significant costs.

I have decided to move this lesson up in the curriculum and teach it when we have school again. School is scheduled to start again on Tuesday, though at this stage everything is provisional, as each hour we find out about more families that have had their homes destroyed by the rising waters. It is still unclear how all our staff, let alone students, will get to school.

I am worried that the lesson could re-traumatize students who have experienced so much trauma in the past few days. I know I will need to make an active effort to make students feel comfortable stepping into the hall if they are feeling overwhelmed. However, my experiences with the recent presidential election make me think that this lesson is exactly what some students might need.

After the election, many students were genuinely confused about what had happened. One question in particular was on their minds: How you can you win the popular vote but not the election? We talked through the Electoral College together, and having clarity about what had happened and why it happened seemed to give them a firmer foundation to build on as they processed their emotions. I am hopeful that teaching about flooding will help ground them in a similar way.

This lesson about flooding was once simply another lesson in the curriculum, but now it has taken on a new urgency. In moments of disaster, it is easy to feel powerless; I certainly could not help the people I saw posting on Facebook that they were been on hold with 911 for hours while standing on their roofs.

Yet teachers have a unique power — the power to shape the minds of future generations to solve the problems that we face. Houston’s location means that it will always be susceptible to flooding. But by teaching about the flood I hope I can play a small role in helping our city avoid repeating some of the tragic scenes I witnessed this week.

Alex McNaughton teaches history and geography at YES Prep Southeast in Houston.

Looking to help? YES Prep is collecting donations to support its students and their families. Houston ISD and KIPP Houston are also soliciting donations for their students.