First Person

Survivorship Bias And The Hidden Costs Of Backfill

Out of 90 charter schools that administered the New York State standardized tests in both 2011 and 2012, Harlem Link had the eighth-highest average increase in English language arts and math scores. This score improvement was amazing, fantastic, even inspiring. And misleading — because of a small, relatively unknown factor called “survivorship bias.”

Survivorship bias is a statistical term for an indication that there is some hidden factor that excludes certain members of a data set over time — namely, part of a sample that was there at the beginning is no longer there at the end and does not count in the final analysis. The smaller subset of those who “survive” over time might be better off than the original whole group simply because of who stayed and who left, not any value added over time.

Simply put, every year, at every school, some students leave, and their departure changes the profile of who takes the test from year to year. Sometimes high-scoring students depart. At other times, low-scoring students depart.

If schools continuously enroll new students (and some don’t), the same factor impacts the student population for these incoming students. At the end of this piece I chart a hypothetical situation in which survivorship bias shows how a school can appear to improve while not actually adding any value simply by not adding new students year after year.

In large systems, there is so much mobility that these student profiles tend to cancel each other out because of scale. For example, the student population appears relatively stable from year to year in the third grade in Community School District 3, where 1,342 students in 30 schools took the state English Language Arts exam in 2012. But in small student populations like the one at Harlem Link, where only 52 third-grade students took the 2012 exam, a few students entering or leaving the school with certain test scores can make a big difference.

When the state department of education releases test scores each year, however, it does not provide this or any other contextual background information alongside the scores. I believe that this process penalizes, in the public eye, schools that continue to enroll students to replace those that depart.

(Partly) illusory gains

At Harlem Link, the fact that we only test in three grades guarantees that at least one third of our students taking the tests each year will be different students from those who took it the year before. Putting aside the variability in the state test from year to year, this rolling of the dice has influenced some dramatic swings in achievement that mean our school’s test scores have looked worse than the actual performance of our teachers in some years, and at other times (like this year) they may have looked better than they really were.

It turns out that the profile of our students who departed before the last school year was a much less successful one than the profile of the group that left the prior year. In other words, we had to improve less to get apparently lofty gains.

In English Language Arts, we saw an improvement of 18 percentage points from 2011 to 2012, according to the state’s way of reporting the scores. But since many of the students who graduated in 2011 or left for other reasons following the 2010-2011 academic year performed poorly on the 2011 exams, the students who returned had a better passing rate by 10 percentage points than the original group. In other words, more than half of our test score gains in ELA could be accounted for by attrition.

Now, I’m not going to say that I’m not proud of our scores or that they are not indicative of a powerful effort by talented and dedicated professionals. I’m not even going to tell you that we didn’t improve our practice last year. I think we have improved in that area every year, because we have been an honest, self-examining, learning organization. But the wide swing in test scores and the state’s failure to describe enrollment patterns when reporting the scores masks the true story of a gradual, continuous march to improvement that is the real hallmark of the growth at Harlem Link.

Best practices often begin as difficult, controversial and seemingly impossible changes to “the way things are.” Strong schools take the time required to plan, assess, and tweak new initiatives until they become standard operating procedures. The lack of information provided alongside scores obscures this type of growth, creating perverse incentives for schools to “push out” students who are low performers and to “quick fix” by whittling down large original cohorts to smaller groups of survivors, uncompromised by new admittees.

At Harlem Link, we have resisted these perverse incentives. We have always replaced students who leave, for budgetary reasons (being a small, standalone charter school) and to serve a greater portion of the community starved for high-quality school choices. Each year, we have encouraged some students who are particularly high-achieving to leave a year early by helping them apply to competitive public and independent middle schools that only admit in fifth grade, reasoning that we’d rather lose their strong fifth-grade test scores than see them lose an opportunity to get firmly on the college track a year ahead of their peers.

If we followed the short-sighted state incentive, we would not have urged four of our highest-scoring fourth-graders on the state exams in 2012 to apply to and enter the Upper West Side’s highly sought-after Center School. They were admitted and are all attending — a fact that may well push down our fifth-grade test scores by as much as 10 percent next year — and we are thrilled, because we helped four more students living in a high-poverty environment to gain admission to this exclusive public school.  We also would not have pushed students leave after fourth grade in years past to embark on the independent school track by attending the East Harlem School at Exodus House and the George Jackson Academy in lower Manhattan.

In the context of reform

This issue has been raised before in the blogosphere, but not in a thoughtful manner. Instead, it has been wielded as a weapon by those who are against the current strain of education reform. It has been used to defeat the straw-man argument that charters are silver bullets and to denigrate the success of networks like KIPP, which is another organization that deserves no such uninformed criticism. (Each year, KIPP asks itself several questions in its annual internal reporting, including, “Are we serving the students who need us?”)

Because it is potentially embarrassing and might burst the balloon of so-called charter education miracles, this issue has also (to my knowledge) been ignored publicly by my colleagues in the charter community. There are many groups of charter schools that go happily on their way winnowing down their large kindergarten classes, educating fewer and fewer students in each cohort each year, not adding new students and narrowing down their challenges as they deal with fewer and fewer “survivor” students well. And those charters that benefit from network infrastructure and economies of scale can balance their budgets even while shrinking six to eight kindergarten sections down to three or four fifth-grade sections.

I’m not passing judgment on those networks. As a charter school founder who has been running a school for almost 10 years, I still believe that the charter experiment has been a profoundly positive one for the communities where such schools have flourished. What I want is for the public to have some understanding of the context behind test scores, so alleged miracles can be put in their proper place, and year-to-year statistical swings that have nothing to do with a school community’s actual performance can be put into their proper perspective.

Hypothetical (with some assumptions): Survivorship bias in action

In the example below, compare two schools that start out with similar student profiles. School A replaces each student who departs. School B does not.

Each year at both schools, a greater percentage of academically struggling students than successful students leave. Each year at both schools, neither school is adding any value since no individual’s test scores are changing.

Because the entering students at School A are similarly academically disadvantaged to those who depart, its scores do not change. School B’s scores improve more than 20 percentage points — simply by virtue of attrition, the decision not to enroll new students, and the mix of which students are taking the test each year.

School A enrolls new students continuously

School B does not enroll new students

Steve Evangelista is the principal of Harlem Link Charter School. This piece originally appeared on the school’s blog.

First Person

I was an attorney representing school districts in contract talks. Here’s why I hope the Supreme Court doesn’t weaken teachers unions.

PHOTO: Creative Commons / supermac1961

Many so-called education reformers argue that collective bargaining — and unions — are obstacles to real change in education. It’s common to hear assertions about how “restrictive” contracts and “recalcitrant” unions put adult interests over children’s.

The underlying message: if union power were minimized and collective bargaining rights weakened or eliminated, school leaders would be able to enact sweeping changes that could disrupt public education’s status quo.

Those that subscribe to this view are eagerly awaiting the Supreme Court’s decision in the case of Janus v. American Federation of State, County, and Municipal Employees. At issue is the constitutionality of “agency” or “fair share” fees — employee payroll deductions that go to local unions, meant to cover the costs of negotiating and implementing a bargaining agreement.

In states that permit agency fees (there are about 20), a teacher may decline to be part of a union but must still pay those fees. If the Supreme Court rules that those agency fees are unconstitutional, and many teachers do not voluntarily pay, local unions will be deprived of resources needed to negotiate and enforce bargaining agreements.

Based on my experience as an attorney representing school districts in bargaining and contract issues, I have this to say to those hoping the Court will strike down these fees: be careful what you wish for.

Eliminating fair share fees (and trying to weaken unions) represents a misguided assumption about bargaining — that the process weakens school quality. To the contrary, strong relationships with unions, built through negotiations, can help create the conditions for student and school success. Indeed, in my experience, the best superintendents and school boards seized bargaining as an opportunity to advance their agenda, and engaged unions as partners whenever possible.

Why, and how, can this work? For one, the process of negotiations provides a forum for school leaders and teachers to hear one another’s concerns and goals. In my experience, this is most effective in districts that adopt “interest-based bargaining,” which encourages problem-solving as starting point for discussions as opposed to viewing bargaining as a zero-sum game.

Interest-based bargaining begins with both sides listing their major concerns and brainstorming solutions. The touchstone for a solution to be adopted in a bargaining agreement: Is the proposal in the best interests of children? This important question, if embedded in the process, forces both sides to carefully consider their shared mission.

For example, some districts I worked with paid teachers less than comparable neighboring districts did. It would have been unreasonable for unions to insist that their pay be increased enough to even that difference out, because that would mean reducing investments in other items of importance to children, like technology or infrastructure. At the same time, it would have been untenable for management to play “hard ball” and deny the problem, because to do so would likely lead to a disgruntled workforce.

Instead, both sides were forced to “own” the issue and collaboratively craft plausible solutions. That made unions more agreeable to proposals that demonstrated some commitment by the district to addressing the issue of pay, and districts open to other things that they could provide without breaking the budget (like more early release days for professional development).

To be sure, many school administrators could get frustrated with the process of bargaining or having to consult the negotiated agreement when they want to make a change. Some districts would very much like to adopt an extended school day, for example, but they know that they must first consult and negotiate such an idea with the union.

Yet, in districts where school administrators had built a reservoir of goodwill through collective bargaining, disagreement does not come at the cost of operating schools efficiently. Both sides come to recognize that while they inevitably will disagree on some things, they can also seek agreement — and often do on high-stakes matters, like teacher evaluations.

How does this relate to the Supreme Court’s pending decision? Without fees from some teachers, unions may lack the resources to ensure that contract negotiations and enforcement are robust and done well. This could create a vicious cycle: teachers who voluntarily pay fees for bargaining in a post-Janus world, assuming the court rules against the unions, will view such payments as not delivering any return on investment. In turn, they will stop contributing voluntarily, further degrading the quality of the union’s services.

Even more troubling, if fair share fees are prohibited, resentment and internal strife will arise between those who continue to pay the fees and those who refuse. This would undercut a primary benefit of bargaining — labor peace and a sense of shared purpose.

Speaking as a parent, this raises a serious concern: who wants to send their child to a school where there is an undercurrent of bitterness between teachers and administrators that will certainly carry over into the classroom?

It is easy to see the appeal of those opposing agency fees. No one wants to see more money going out of their paycheck. The union-as-bogeyman mentality is pervasive. Moreover, in my experience, some teachers (especially the newer ones) do not recognize the hidden benefits to bargaining contracts.

But, obvious or not, agency fees help promote a stable workplace that allows teachers to concentrate on their primary responsibility: their students. Removing the key ingredient threatens this balance.

Mark Paige is a former school teacher and school law attorney who represented school districts in New England. He is currently an associate professor of public policy at the University of Massachusetts – Dartmouth.

First Person

I’m a Florida teacher in the era of school shootings. This is the terrifying reality of my classroom during a lockdown drill.

Outside of Marjory Stoneman Douglas High School in Parkland, Florida. (Photo by Mark Wilson/Getty Images)

“Remember,” I tell the children, looking them in the eyes in the darkened classroom. “Remember to keep the scissors open. They’ll stab better that way.”

My students, the target demographic for many a Disney Channel sitcom, laugh nervously at me as they try to go back to their conversations. I stare at the talkative tweens huddling in a corner and sigh.

“Seriously, class,” I say in the tone that teachers use to make goosebumps rise. As they turn back to me with nervous laughter, I hold up that much-maligned classroom tool, the metal scissor that’s completely ineffective at cutting paper. “If a gunman breaks in, I’ll be in the opposite corner with the utility knife.” Said tool is in my hand, and more often used to cut cardboard for projects. All the blood it’s hitherto tasted has been accidental. “If I distract him and you can’t get out, we have to rush him.” I don’t mention that my classroom is basically an inescapable choke point. It is the barrel. We are the fish.

They lapse into silence, sitting between the wires under the corner computer tables. I return to my corner, sidestepping a pile of marbles I’ve poured out as a first line of defense, staring at the classroom door. It’s been two hours of this interminable lockdown. This can’t be a drill, but no information will be forthcoming until it’s all over.

I wonder if I really believe these actions would do anything, or am I just perpetrating upon my students and myself the 21st century version of those old “Duck and Cover” posters.

We wait.

The lockdown eventually ends. I file it away in the back of my head like the others. Scissors are handed back with apathy, as if we were just cutting out paper continents for a plate tectonics lab. The tool and marbles go back into the engineering closet. And then, this Wednesday, the unreal urge to arm myself in my classroom comes back. A live feed on the television shows students streaming out of Marjory Stoneman Douglas, a high school just a short drive away. I wonder whether the teachers in its classrooms have passed out scissors.

*

The weapons. It’s not a subject we teachers enjoy bringing up. You’d have an easier time starting a discussion on religion or politics in the teacher’s lounge then asking how we all prepare for the darkness of the lockdown. Do you try to make everyone cower, maybe rely on prayer? Perhaps you always try to convince yourself it’s a drill. Maybe you just assume that, if a gun comes through the door, your ticket is well and truly up. Whatever token preparation you make, if at all, once belonged only to the secret corners of your own soul.

In the aftermath of Parkland, teachers across the nation are starting to speak. The experience of being isolated, uninformed, and responsible for the lives of dozens of children is now universal to our profession, whether because of actual emergencies or planned drills. You don’t usually learn which is which until at least an hour and sometimes not until afterwards. In both cases, the struggle to control the dread and keep wearing the mask of bravery for your students is the same.

And you need a weapon.

I’ve heard of everything from broken chair legs lying around that never seem to be thrown away to metal baseball bats provided by administration. One teacher from another district dealt with it by always keeping a screwdriver on her desk. “For construction projects,” she told students. She taught English.

There’s always talk, half-jokingly (and less than that, lately) from people who want teachers armed. I have a friend in a position that far outranks my own whose resignation letter is ready for the day teachers are allowed to carry guns in the classroom.

I mean, we’ve all known teachers who’ve had their cell phones stolen by students …

*

Years earlier, I am in the same corner. I am more naïve, the most soul-shaking of American massacres still yet to come. The corner is a mess of cardboard boxes gathered for class projects, and one of them is big enough for several students to crawl inside.

One girl is crying, her friend hugging her as she shakes. She’s a sensitive girl; a religious disagreement between her friends having once brought her to tears. “How can they be so cruel to each other?” She asked me after one had said that Catholics didn’t count as Christians.

I frown. It’s really my fault. An offhand comment on how the kids needed to quiet down because I’m not ready to die pushed her too far. Seriously rolling mortality around in her head, she wanted nothing more than to call her family. None of them are allowed to touch their cell phones, however, and the reasoning makes sense to me. The last thing we need is a mob of terrified parents pouring onto campus if someone’s looking to pad their body count.

She has to go to the bathroom, and there are no good options.

I sit with her, trying to comfort her, wondering what the occasion is. Is there a shooter? Maybe a rumor has circulated online. Possibly there’s just a fleeing criminal with a gun at large and headed into our area. Keeping watch with a room full of potential hostages, I wonder if I can risk letting her crawl through the inner building corridors until she reaches a teacher’s bathroom. We wait together.

It seemed different when I was a teen. In those brighter pre-Columbine times, the idea of a school shooting was unreal to me, just the plot of that one Richard Bachman book that never seemed to show up in used book stores. I hadn’t known back then that Bachman (really Stephen King) had it pulled from circulation after it’d been found in a real school shooter’s locker.

Back then my high school had plenty of bomb threats, but they were a joke. We’d all march out around the flagpole, sitting laughably close to the school, and enjoy the break. Inevitably, we’d all learn that the threat had been called in by a student in the grip of “senioritis,” a seemingly incurable disease that removes the victim’s desire to work. We’d sit and chat and smile and never for a second consider that any of us could be in physical danger. The only threat we faced while waiting was boredom.

*

Today, in our new era of mass shootings, the school districts do what they can, trying to plan comprehensively for a situation too insane to grasp. Law enforcement officials lecture the faculty yearly, giving well-rehearsed speeches on procedures while including a litany of horrors meant to teach by example.

At this level, we can only react to the horrors of the world. The power to alter things is given to legislators and representatives who’ve been entrusted with the responsibility to govern wisely while listening to the will of the people. It’s they who can change the facts on the ground, enact new laws, and examine existing regulations. They can work toward a world where a lockdown is no longer needed for a preteen to grapple with gut-churning fear.

We’re still waiting.

K.T. Katzmann is a teacher in Broward County, Florida. This piece first appeared on The Trace, a nonprofit news site focused on gun violence.