First Person

Bungled by Design

There’s a well-known education research textbook by three distinguished scholars at Harvard entitled By Design.  Judy Singer, one of the authors, once told me that the working title for the book, rejected by Harvard University Press, was Bungled by Design.  That title conveyed the key message of the book, which is that, when it comes to education research, you can’t fix by analysis what you bungled by design.  The design of a research study dictates what a researcher can plausibly ask, and the credibility of the claims about what is being studied.

The recently-released NYU study of the New York City Principal Leadership Academy comparing graduates of the Aspiring Principals Program to other new NYC principals is, in my view, bungled by design.  This is not a knock on the authors, each of whom I know and respect a great deal.  Rather, it reflects the fact that the NYU researchers were brought in to study the Aspiring Principals Program of the Leadership Academy long after critical design decisions about how to evaluate the impact of the program were made—either by omission or commission.

The three key limitations I raise here pertain to selection mechanisms that ideally would have been observed by the researchers.  The inability to understand and model these selection processes undermines the objective of isolating the effect of the Aspiring Principals Program on student outcomes.  (See the comments of Sean Corcoran, lead author of the report, on selection issues here.)

Let’s start with selection into the program itself.  We know precious little about either the individuals selected into the Aspiring Principals Program or those who self-select into becoming principals in the comparison group.  To be sure, they can be compared on race, age, years of teaching experience, and a couple of other variables, but there’s no information about the personal qualities of the APP and comparison principals—and these personal qualities might be relevant to their subsequent success as principals.  This concern is heightened by the fact that the Aspiring Principals Program is highly selective.  For example, the 2003 cohort of Aspiring Principals consisted of 90 individuals culled from 400 applications;  the number of applications ballooned to 1,200 in 2004, which means that fewer than one in 10 applicants was selected for the program.  The analysis cannot even rule out the uncomfortable possibility that some of the members of the comparison principal group applied to the APP and were rejected because the program administrators predicted that they would be unsuccessful.  There’s really no way to establish that the APP and comparison principal groups were equivalent on things that might matter for their success at the time that the groups were selected.

Next, there’s the selection of the schools that APP and comparison principals were chosen to lead.  The NYU report documents that APP principals were placed in different kinds of schools than comparison principals—schools that were smaller, lower-performing, and on a downward trajectory, with higher concentrations of Black students, and more likely to be located in the Bronx.  Other than the stated mission of the Leadership Academy to place APP principals in hard-to-staff schools, we have little to go on to document the process of assigning principals to schools.  The fact that APP and comparison principals wound up in different kinds of schools greatly complicates an understanding of the impact of the program, because there is a risk that the differences in outcomes which are observed are as much a function of the features of the schools in which principals were placed as the changes in principals’ behaviors attributable to exposure to the APP.  The NYU researchers do the best they can to address this by looking at the performance trajectories of schools pre- and post-arrival of the APP and comparison principals, but there are many unanswered questions.  Suppose, for example, that a school was on a downward trajectory for two years before an APP principal took over.  Might we expect such a school to trend upward just by chance, given what we know about year-to-year fluctuations in school-level test performance?  If so, the slight advances observed in schools led by APP graduates relative to comparison principals might be due to differences in the schools they led, and not to the impact of the APP.  The question here is whether there is a sufficient number of schools led by APP graduates and schools led by comparison principals with similar trajectories prior to the arrival of the new principal to rule out this possibility.             

And third, there’s the issue of selection out of both the APP group and the comparison principal group—what is often referred to as sample attrition.  Of the 147 graduates of the APP in the 2004 and 2005 cohorts (82% of the approximately 180 entrants to the program in those two years), 120 served as a principal for some time in a DOE school.  Of these, 15 switched schools, a few served as a principal and transferred to another DOE position, a couple were promoted, and some left the DOE after serving as a principal.  There are 86 APP graduates of the original 147, or 59% of the graduates, included in the analysis.

The study does not address the attrition either from the APP group or from the comparison group.  (We know next to nothing about the comparison principals who started and didn’t persist in the same school for three years, and thus were vanquished from the study.)  Were the 34 APP graduates who started as principals but didn’t meet the three-year tenure requirement of the study unsuccessful and counseled out (or kicked upstairs)?  How did they differ from the 86 APP graduates who represent the core sample analyzed in the study?  Understanding the impact of a program requires an understanding of who leaves the “treatment” group as well as who leaves the comparison group, and why.  

Finally, I can’t end this post without commenting on the limits of assessing the Aspiring Principals Program primarily on the basis of the state test scores achieved by students (for the elementary/middle schools).  The fact that such test scores are often available—although never for students below grade 3!—and are at the center of the city’s accountability system does not justify the decision to exclude virtually every other measure of principal performance that might be relevant.  Does the principal support students’ social and emotional development?  Or preparation for citizenship in our complex democracy?  Does s/he support the teachers’ learning, and ambitious teaching practices?  Does s/he promote collaborative problem-solving among the staff and stakeholders of the school?  Does s/he manage resources efficiently to support the instructional mission of the school?  Does s/he act with integrity?  These are just some of the desirable features of a skilled principal that the NYU evaluation of the Aspiring Principals Program was unable to address.  Through no fault of the researchers.

And finally-finally:  Although clearly beyond the purview of the NYU project, I am hoping that someone is engaged in a cost-effectiveness analysis of the Aspiring Principals Program.  There’s considerable attrition at various stages up to the desired outcome of a sustained stint as an NYC principal, and the results obtained in the NYU study do not show that students in schools led by an APP graduate have much better outcomes than students in schools led by comparison principals.  A careful cost-effectiveness analysis would juxtapose the student outcomes observed in the two groups with the cost of preparing individuals in the two groups.  There may well be direct and indirect costs associated with preparing principals who do not go through the APP.  We know that the costs of taking an individual through the APP are substantial, perhaps in excess of $150,000.  Do the small and mixed differences in student outcomes favoring APP graduates observed in the NYU study justify investments of this magnitude?

First Person

A Chalkbeat roundtable: The promise and perils of charter networks like Success Academy

When we published an essay about the promise and perils of charter schools by our CEO and editor in chief Elizabeth Green last month, we heard from a lot of readers.

Elizabeth’s piece outlined her conclusions after more than a decade of reporting about charter school networks, and more specifically the Success Academy network in New York City. She wrote that charter school networks offer both great advantages — in their ability to provide rare coherence in what is taught across classrooms — and significant danger. Charter networks, she wrote, have changed public education by “extracting it from democracy as we know it.”

Some of our readers saw their own thinking reflected in her conclusions. Others had a very different take.

What was clear was that Elizabeth had kicked off a conversation that many Chalkbeat readers are ready to have, and that, as always, robust and respectful debate is good for everyone’s thinking.

So we reached out to people who engage with big questions about how schools are structured every single day, in their work or personal lives. Today, we’re sharing what they had to say. But we think this is far from the end of the conversation. If you want to add your voice, let us know.

 
 

 

Charter networks’ needs and goals may not be the community’s

By Tim Ware, former executive director of the Achievement Schools managed by the Tennessee Department of Education and founder of Ware Consulting Group

As the founder and former executive director of a high performing public charter middle school in Memphis, Tennessee, I am a firm believer in the promise of well-run charter schools. I also understand the limits of these schools.

A key aspect of public charter legislation is autonomy. This means that public charters decide how to staff their schools, which curriculum to use, how to allocate resources for student support, and how their daily and summer schedules work. However, this legislated autonomy creates issues that thoughtful policymakers need to address.

For instance, in Memphis, a high-performing public charter network began operating a chronically underperforming middle school as a part of a turnaround intervention effort. Despite significant improvements in learning and school culture, as well as the support of the community, the school grappled with dwindling enrollment and suffocating building maintenance costs. Fewer dollars were available to invest in high quality teaching and learning, social-emotional supports, and extracurricular activities. Ultimately, the charter operator made the difficult decision to cease operating the school.

This example illustrates the limits of public charter schools. The same autonomy that allowed them to create an approach that drove improvement for children also allowed them to decide that they could no longer operate the school. This means that, as long as autonomy exists for public charter schools (and it should), we cannot eliminate traditional districts.

The solution for historically underserved communities will be found by creating strong ecosystems of education. These ecosystems should consist of a healthy mix of traditional schools, optional schools (schools with competitive entry requirements), magnet schools, public charter schools, and private schools. By ensuring that multiple types of schools flourish and are accessible to all, parents will be able to make informed choices and select a school which best meets the needs of their most precious belonging — their child.

Back to top

 
 

 

Focusing on charter networks is a mistake. Districts have the same potential

By Josh Thomases, dean of innovation, policy, and research at Bank Street College of Education

Elizabeth Green’s article on Eva Moskowitz misses one important detail – districts have successfully scaled change for students. In this era of attacks on government, it is worth looking closer.

The hundreds of new small high schools opened in New York City between 2000 and 2012 transformed thousands of lives. The research firm MDRC documented that impact, showing a 9.4 percent increased graduation rate and an 8 percent increase in college attendance. Notably, this increase was driven by success with groups that school systems often fail: poorer students, black students, and students with disabilities.

This extraordinary effort happened with district educators and unions, public resources and processes.

I saw this reform inside and out. I helped create a small school in the 1990s and was part of community protests against some of the initial school closures under Chancellor Joel Klein. And, in 2004, I became responsible for the development and support of new schools within the education department.

The new schools work was an example of democracy in action – with all its imperfections. There were legendary protests against the Department of Education and arguments over race, equity and power. And through all of that, the process transformed schools.

Why the success?

  1. The point was to improve teaching and learning. Everything was looked at through this lens.
  2. Educators were the agents of change. The new schools process challenged principals, teachers, community members and parents to reimagine school.
  3. External partners multiplied the power of the changes. These included school development organizations (such as New Visions and CUNY) and local partners ranging from the Brooklyn Cyclones and South Bronx Churches. For the first six years of the reform, the unions were a partner, too.
  4. The district shifted authority towards the principal and school based staff in key areas: hiring, scheduling, budgets, and curriculum.

This is not a story of perfect success; as a district, we made mistakes and they were debated publicly. But the results show that districts can take bold action to change what is happening in schools.

Charters in New York have also demonstrated they can make an important contribution to a district. The task ahead is not to forego government, but to activate its strengths.

Back to top

 
 
 

 

Charter networks are a laboratory for consistent and high-quality instruction

By Seneca Rosenberg, chief academic officer at Valor Collegiate Academies in Nashville, Tennessee

My first year in the classroom, I desperately wanted to be the teacher my fourth graders deserved. A diligent student, I carefully examined California’s standards, the curriculum my district had adopted, new research, and popular trade books. I quickly saw that the approaches they outlined — for how to teach reading, for example — were often in direct conflict.

Veteran teachers advised: have your students fill out the mandated worksheets to avoid scrutiny, then close your door and teach as you want. This would have been good advice if only I had known what to do behind that door to help my students to learn.

Now, as chief academic officer of Valor Collegiate Academies, a small charter school network in Nashville, I reflect daily on how our autonomy and network structure provide crucial, and often unremarked upon, resources for developing coherent systems of teaching and learning.

Like other charter networks, Valor has the flexibility to set our educational vision and then organize our own curriculum, assessments, hiring policies, student and teacher schedules, and culture to realize it. Many of our teachers and school leaders report that our shared systems, while demanding, buffer them from some of the stress that comes with making sense of dissonant policies and practices they more regularly encountered in traditional public schools.

Even more importantly, our infrastructure provides our teachers and leaders with a common framework around which expertise can be developed, shared, and improved.

For example, at Valor, our teaching teams meet frequently to study and plan from our students’ work. We have shared protocols for data analysis and teacher coaching. Each piece has been intentionally developed as part of a system. As a result, teachers have opportunities to learn that far exceed anything I had access to as a teacher — and our students benefit.

I share some of Elizabeth Green’s ambivalence about the potential impact of the rise of charters nationally, though she inflates the extent to which charters “extract” public education from democratic control — at least in states in which authorizing laws are well crafted. I am also skeptical of Moskowitz’s suggestion that perhaps “a public school system consisting principally of charter schools would be an improvement.”

But charter networks’ unique conditions do provide a useful laboratory. Critics who dismiss our high-performing charter networks’ many successes risk missing what we are learning from this critical innovation — coherent instructional systems — and how that might contribute to new possibilities for American education.

Back to top

 
 
 

 

In my city, no schools have it figured out

By Bernita Bradley, parent advocate and blogger at Detroit School Talk (and a Chalkbeat Reader Advisory Board Member)

Take all kids out of charter schools, they say. Close them down and require those students to attend their closest public school, no matter how far, how full the classrooms, and how low-performing. Hop on a bus more than 25 minutes to attend the closest high school near you and sit at the back of the class on the floor. After all, public schools were perfect before charter schools came along, and in order for them to be perfect again, we need everyone on board.

Don’t talk bad about public schools, they say. Don’t draw attention to the fact that we are still figuring out how to improve public schools and need your help. The city of Detroit must unite, be of one mind, and let all charter school leaders know that we are only supporting traditional public schools.

These arguments won’t work. I fight for quality public schools and fought for us to not lose more of them. However, if you strip parents of choice, you prove that you are not committed to providing children with what they need.

To be clear, I am an advocate for both sides. Parents don’t care about this war — we just want good schools that will educate all children equally. Can we have that conversation?

Let’s tell the truth about how, here in Detroit, both sides cherry-pick students and “counsel out” parents. Public schools just suspend students indefinitely until parents leave to find a charter school. Let’s tell the truth about how teaching to the test has affected both charter and public school teachers’ ability to make sure student academic growth is more robust.

Both sides could do better. My children have attended both kinds of schools. I’ve bused my kids 15 miles away. I’ve sent my kids to the top charter and public schools in the city. And no one — including charter schools — has this figured out.

I can’t think of a person would say they are totally happy with their child’s educational experience here in Detroit. We have come to the point where, while we’ve made friends in both charters and public schools, this is a journey full of struggles and broken promises that we would not wish on any parent.

Believe me, if we had our way there would be no need to choose. The school on the corner would be full and alive with students, parents, and teachers who have one common goal, to educate all kids.

Back to top

 
 
 

The rise of networks hurts the charter movement

By Steve Zimmerman, Coalition of Community Charter Schools

In the ongoing saga of Eva Moskowitz and her war against the the educational status quo, two key issues are overlooked. The first is that the rise of Success Academy has come at significant cost to the charter school movement and the democratic values that were at its genesis.

The rigidly top-down managerial approach of the Success network is the antithesis of the original idea of chartering: to free schools from district-imposed conformity so they have autonomy to innovate. There is no autonomy or innovation in a franchise. Franchisees follow the script.

The second issue is that Success Academy schools, for all intents and purposes, turn teachers into technicians. They are trained in a rigid model of classroom management with a relentless focus on student outcomes. As Elizabeth Green and others point out, the effectiveness of this system, at least in terms of test scores, is well documented and ostensibly justifies the orthodoxy of “no excuses” education reform.

Relentlessness, however, comes at a cost. Just as legendary as its record-high test scores is Success Academy’s teacher attrition. Success Academy appears to welcome an increasing number of bright young people to learn and execute the scripts, and then watch as they move on to their real careers after they burn out in three years. The consequences of this trend are chilling to imagine.

If we believe the purpose of public education to be the development of exceptional test takers, then Eva Moskowitz has clearly pointed the way to the promised land. If, however, we believe the purpose is the betterment of society and the development of the whole child, there are better models to emulate.

Back to top

 
 
 

 

Coherence is important, but charter networks aren’t necessary to achieve it

Andy Snyder, social studies teacher at Harvest Collegiate High School in New York City

Who should decide what students learn in school? Families or individual teachers? District and charter school leaders, elected officials, or panels of professors?

Elizabeth Green’s recent essay focuses our attention on this huge question. She points out that many other countries provide “a clear sense of what students need to learn, the basic materials necessary to help them learn it (such as a curriculum).” And she argues that some charter school networks, enabled by their anti-democratic powers, are developing coherent and meaningful ideas of what to prioritize and how to teach it well.

When I began student teaching, I was shown stacks of textbooks and boxes of transparencies, quizzes, tests, homework — corporate-branded, filled with facts, empty of meaning. I switched to another mentor and recreated the trial of John Brown. Later I left one innovative public school where administrators were attempting to bend my courses into more traditional shapes for another where the interview includes, “Describe a dream course that you would love to teach” and where we teach those courses every day.

But I’ve seen in Germany the effects of a thoughtful curriculum — classes connect between disciplines and spiral powerfully between grades, and teachers adapt rather than invent.  Improvised individual efforts often produce a worse result than a strong system. That’s why I commute in New York by subway, not bicycle.

The systemic approach can break down too. Today we curse the defunding of our transit agency, and we saw what happened to the Common Core. How can charter schools develop truly excellent curriculum when their priority seems to be preparing students to win against bad bubble tests?

Students, no matter what kind of school they attend, deserve lessons crafted by well-trained practitioners who draw from the best ideas of the profession.

In the best future I can imagine, each school or district adapts curriculum from one of several coherent curriculum packages developed over years with millions of dollars and genius and honest sweat. Teachers trained in that tradition lead students in cultivating the deep questions and necessary knowledge, and students graduate with a sense of how it all adds up and what they can bring with them into the world.

Back to top

First Person

I’m a teacher in Memphis, and I know ‘grading floors’ aren’t a cheat — they’re a key motivator

PHOTO: Creative Commons / Shelly

Growing up, my father used to tell me not to come to him with a problem unless I had a solution.

That meant I learned quickly what kinds of solutions wouldn’t go over well — like ones involving my father and his money. His policy also meant that I had to weigh pros and cons, thinking about what I was able to do, what I wasn’t, and whom I needed help from in order to make things happen.

I sometimes wish decision-makers in Memphis had a father like mine. Because more often than not, it seems we are talking about the problems void of a solution or even possible solutions to vet.

Right now, the issue in Memphis and Shelby County Schools is the “grading floor,” or the policy of setting a lowest possible grade a teacher can assign a student. They have been temporarily banned after a controversy over high-school grade changing.

Grading floors aren’t new to teachers in Memphis, or to me, a fifth-grade teacher. I have taught and still teach students who are at least two grade levels behind. This was true when I taught fourth grade and when I taught sixth grade. Honestly, as the grade level increased, so did the gaps I saw.

More often than not, these students have been failed by a school, teacher, leader or system that did not adequately prepare them for the next grade. Meanwhile, in my classroom, I have a responsibility to teach grade-level material — adjusting it for individual students — and to grade their work accordingly.

That’s where “grading floors” come in. Without a grading floor, all of my current students would have grades below a 65 percent.

Can you imagine seeing the face of a fifth-grade boy who tried his hardest on your test, who answered all the questions you gave orally, who made connections to the text through auditory comprehension, only to receive a 0 on his paper?

I don’t have to imagine – I see similar reactions multiple times a day. Whether it’s a 65 percent or a 14 percent, it’s still an F, which signals to them “failure.” The difference between the two was summed up by Superintendent Hopson, who stated, “With a zero, it’s impossible to pass a course. It creates kids who don’t have hope, disciplinary issues; that creates a really bad scenario.”

I know that as years go by and a student’s proficiency gap increases, confidence decreases, too. With a lowered confidence comes a lower level of self-efficacy — the belief that they can do what they need to do to succeed. This, to me, is the argument for the grading floor.

In completing research for my master’s degree, I studied the correlation between reading comprehension scores and the use of a motivational curriculum. There was, as might have guessed, an increase in reading scores for students who received this additional curriculum.

So every day, I speak life into my students, who see Fs far too often in their daily lives. It is not my job as their teacher to eradicate their confidence, stifle their effort, and diminish their confidence by giving them “true” Fs.

“This is not an indication of your hard work, son. Yet, the reality is, we have to work harder,” I tell students. “We have to grind in order to make up what we’ve missed and I’m the best coach you have this year.”

In education, there are no absolutes, so I don’t propose implementing grading floors across the board. But I do understand their potential — not to make students appear more skilled than they are, or to make schools appear to be better than they are, but to keep students motivated enough to stay on track, even when it’s difficult.

If it is implemented, a grade floor must be coupled with data and other reports that provide parents, teachers, and other stakeholders with information that accurately highlights where a student is, both within the district and nationally. Parents shouldn’t see their child’s progress through rose-colored glasses, or be slapped by reality when options for their child are limited during and after high school.

But without hope, effort and attainment are impossible. If we can’t give hope to our kids, what are we here for?

I don’t have all the answers, but in the spirit of my father, don’t come with a problem unless you have a solution.

Marlena Little is a fifth-grade teacher in Memphis. A version of this piece first appeared on Memphis K-12, a blog for parents and students.