School choices

School choice supporters downplay new voucher research, saying schools are more than a test score

PHOTO: Creative Commons / Michael Vadon
Former Florida Governor Jeb Bush.

At this week’s gathering of school choice supporters, there was an awkward fact in their midst: A wave of new studies had shown that students receiving a voucher did worse, sometimes much worse, on standardized tests.

That was the inconvenient verdict of studies examining programs in Louisiana, Ohio, Washington, D.C., and in Indianapolis, where the advocates had convened for the annual conference of the American Federation for Children. U.S. Secretary of Education Betsy DeVos, the group’s former leader, gave the keynote address.

But many of the school choice proponents, who had long made the case that their favored reform works, had an explanation at the ready.

Jeb Bush, the former Florida governor, only alluded to the recent studies. “In spite of a few research projects of a narrowly identified group of students, the simple fact is when you create a marketplace of choices and informed parents … the children do better,” he told the audience.

Other leading supporters emphasized the impact the programs have beyond test scores, as well as the shortcomings of recent studies.

“Some of the data that is really interesting [looks at] not just achievement, but attainment,” Robert Enlow, head of EdChoice, a group that backs vouchers and tax credit programs, told Chalkbeat. “A kid may not be doing as well on a test score as we would like, but they’re graduating at higher rates [and] they’re going into college at higher rates.”

Indeed, older studies show that students in Milwaukee’s voucher program were more likely to graduate high school and enroll in college. Students in D.C.’s initiative also completed high school at a higher rate.

Enlow also pointed to evidence that private school choice can spur improvements in public schools through competition and increase parent satisfaction rates. Sounding a bit like some of his opponents who lead teachers unions, Enlow argued that test scores are a poor measure of educational quality.

“We want a vibrant society of people who know what they’re doing who are productive members of society,” he said. “A single test doesn’t prove jack about that.”

While EdChoice has said that school choice leads to academic gains, the group has also argued, prior to the recent studies, that parents care about more than just test scores when choosing schools. EdChoice opposes requiring students in voucher programs to take state tests at all. Without such data, making comparisons to public schools is more difficult.

Still, Enlow said, “there are some studies showing that private schools need to get better on test scores.”

Supporters also noted that the studies in D.C. and Louisiana were based on just one and two years of data, respectively. Enlow says that is too little information to draw helpful conclusions, a point echoed by Kevin Chavous, a board member at the American Federation for Children and a former D.C. city council member.

“This is after one year in the program,” said Chavous referring to the recent D.C. report, which analyzed three groups of students after a single year of receiving a voucher. “Studies also show … the longer the kids are in these programs, the better they’ll do.”

An overview of past research on school vouchers, including studies in other countries, found that students were neither helped nor harmed after three years, but saw significant test score jumps in the fourth year.

DeVos hasn’t addressed the topic in depth. After her own Department of Education released the report on the D.C. Opportunity Scholarship Program, DeVos stated, “The study released today found that D.C. OSP parents overwhelmingly support this program, and that, at the same time, these schools need to improve upon how they serve some of D.C.’s most vulnerable students.”

Chavous argues that giving families choice means allowing them to pick schools based on what is important to them, which may not be test scores. It’s also hypocritical for those who are skeptical of testing to then use test results to criticize voucher programs, he said.

“You can’t have it both ways — you can’t say we have too much high-stakes testing when it comes to public schools and then when it comes to private choice programs, OK, they aren’t passing the test,” he said.

But he acknowledges inconsistency on his own side among those who use test results to claim that public schools are failing.

“We’re all hypocrites on the testing thing,” Chavous said.

This story has been updated to clarify EdChoice’s previous statements on the value of test scores.

Roll call!

What states told Chalkbeat about how they will monitor their chronic absenteeism data

PHOTO: Erin Einhorn

Now that students’ rates of chronic absenteeism are being used to judge schools in most states, there will be new incentives to manipulate this data. So Chalkbeat asked the 10 state departments of education whose approved ESSA plans use chronic absenteeism how they plan to guard against that. Nine got back to us. Here’s what they said.

Arizona

As with all school attendance data our School Finance and accounting department have certain controls in place to make sure reporting is accurate – since state funding for schools is based in AZ on average daily attendance it is very important.

As for our newly approved ESSA plan, the Department is meeting with various stakeholders to determine ways to improve reporting of attendance and absenteeism. Our goal is of course to try to reach students that are frequently absent and hopefully help get them back in the classroom.

— Dan Godzich, Arizona Department of Education

Connecticut

Connecticut has been collecting and reporting chronic absenteeism data for many years. The data collection system has many in-built edits checks to ensure data quality.

Here are some examples:

  • Attendance must be submitted for all students who are enrolled in the district.
  • Every student with perfect attendance is flagged for district review.
  • Any school that has an increase or decrease of chronic absenteeism rates of more than five percent is flagged for district review and response to [Connecticut State Department of Education].

In addition to edit checks, districts are provided with numerous attendance reports to review their data prior to finalization. Ultimately, the superintendent is required to certify that their data are accurate. Staff also monitor the data during the collection process and reach out to districts when there are anomalies.

Our accountability system is not intended to be a “gotcha.” It’s much more a resource to inform improvement. From the state perspective, we are not looking to name/shame people but partner with them and bring the requisite support to collectively problem-solve with them, which would give districts much less of a reason to cheat or manipulate the data.

— Laura Stefon, Connecticut State Department of Education

Delaware

Delaware school districts and charter schools have been collecting and reporting data regarding attendance as well as absences for many years, so this is not necessarily a new data point in our system. The Department is reviewing current processes around absenteeism and chronic absenteeism, and will work with districts and schools to determine consistent rules for using this information in the accountability system.

— Susan Haberstroh, Delaware Department of Education

Illinois

Historically we have collected school level absence data in summary (e.g. this school had x total absences and x number of students who were chronically truant). Starting this year, 2017-18, we are collecting student-level absence data. As with most of the data we collect from districts, it’s self-reported. We will only know about the absences they report. There are audits and data quality procedures we utilize in order to make sure the data is as accurate possible, for example, a student can’t have more absences recorded than days enrolled in the school (how can you have 30 absences if you’ve only been enrolled for 20 days). In addition, several times a year staff from the Data Analysis division attend conferences and answer questions on a variety of [Illinois State Board of Education] data collection systems … Those dialogues produce suggestions directly from districts that influence changes we make to those systems to improve usability and data accuracy.

— Jackie Rodgers, Illinois State Board of Education

Maine

Maine is planning on using chronic absenteeism and looking at students who have absences of more than 10% of the school year. Student level attendance data must be submitted and “reviewed” quarterly by school districts. The “reviewed” means that they state that they have submitted attendance data to date. At the end of the school year, superintendents will be required to certify that the data is complete and accurate. In addition, one of the duties of the new ESSA Data Coordinator, a new position at the Maine DOE, will be to monitor this data. The exact procedure and policy around this is in process of being written.

— Rachel Paling, Maine Department of Education

Massachusetts

Our system collects attendance data directly from the district student information management systems on a daily basis and also receives any changes made to the student record on a daily basis. So any data manipulation would need to be done systematically and could not be done after the fact without raising flags as to why so many post-dated attendance changes were being made. That being said, it’s something we’ll monitor closely.

— Jacqueline Reis, Massachusetts Department of Elementary and Secondary Education

Oregon

Districts submit attendance data for individual students to [the Oregon Department of Education] four times a year. These data are submitted according to rules published in our data collection manuals; rules that conform to State Board adopted administrative rules and to state statute.

Once the data is loaded into ODE’s data system, ODE staff evaluate the data (such as comparing to historical trends) in order to identify unusual aggregated data, unusual shifts in aggregate trend data, and the plausibility of the individual student level data. (This is part of our usual data quality assurance process, which we use to help ensure data is as accurate as possible.) We know that accidental data submission errors do occur, so our process is designed to help find and correct these.

Once we see data with an unusual pattern we notify the affected districts and ask them to review their submitted data. In some cases districts confirm the data as accurate, in other cases they realize they’ve need to correct an (unintentional) data submission error.

We have not yet done any audits of local district data on attendance. However, the accountability office has conducted such audits on other accountability data. I would anticipate that we might do the same with assessment data submissions, should concerns arise regarding the validity of that data.

— Jon Wiens, Oregon Department of Education

New Jersey

New Jersey has collected chronic absenteeism data for many years. In the School Performance Reports, New Jersey has included chronic absenteeism data for elementary and middle schools for years, and beginning this last year, in the 2016 School Performance Reports, chronic absenteeism data was included for high schools. By publicly posting the chronic absenteeism data for each school on the [New Jersey Department of Education] website, the public is able to have conversations with schools about their data.

— David Saenz, New Jersey Department of Education

Tennessee

There are checks in place to ensure that our large-scale data collection processes are as accurate as possible. In addition to our auditing and compliance processes, which ensure districts are following proper procedures, the department also collects data separately through our assessment and evaluation process, and that provides another check. Teachers claim the students who have been present for the vast majority of instructional time for the purposes of their evaluation. If this data does not align with chronic absenteeism rates reported for the school, it would indicate something was off, and we could investigate further. Additionally, teachers and administrators have another incentive to mark attendance data correctly. If a student is marked as present but is really absent, there could be liability issues for the school. For example, if the school said a student was in class all day but got in an accident or committed a crime during that window, the school could be liable. Liability is a concern we hear from districts, so we know this is on the minds of many administrators.

— Sara Gast, Tennessee Department of Education

Taking attendance

Student absences are about to have higher stakes in most states. Will cheating follow?

PHOTO: RJ Sangosti, Denver Post

Schools across the country are about to be held accountable for student attendance — attaching stakes to a measure that previously had much less significance and increasing the risk that schools will try to manipulate that data.

But it’s unclear how effectively states have prepared for that possibility, or have systems in place to accurately monitor absenteeism data at all.

“It’s human nature, when the stakes rise, to want to game the system,” said Phyllis Jordan of the Georgetown-based think tank FutureEd. She recently wrote an analysis finding that 36 states plan to use chronic absenteeism to measure schools under ESSA, the federal education law. “In that regard, I don’t think chronic absenteeism is any different than other measures, like test scores.”

Of course, one way for schools to improve their chronic absenteeism marks is to add support that helps students to show up to school. That’s exactly how experts and policymakers hope educators will respond, and because states are only using chronic absenteeism as a small portion of the accountability system, the incentives for cheating may not be strong. But past experience with evaluation systems suggests that a small number of schools will resort to unscrupulous means.

“When you’ve got high stakes on something, if there’s a way to corrupt it, some people are going to corrupt it,” said Diane Whitmore Schanzenbach, an education researcher at Northwestern who wrote a brief highlighting the advantages of using chronic absenteeism to measure schools. “The question is, how big is this incentive? How many schools are going to engage in this bad behavior?”

High stakes could lead to manipulation, but how big of a problem this will be is an open question

A 2003 study in Chicago found evidence of cheating on standardized tests in about 5 percent of elementary classrooms. More subtle gaming has also occured: research found evidence that teachers focused on topics likely to appear on the state test, at the expense of other academic standards.

The potential problem may be more acute when it comes to student absences because of the all-or-nothing way chronic absenteeism is measured.

In most states, a student is deemed chronically absent if they miss 10 percent of school days — around 18 days for those enrolled for a full school year. That means that schools might be especially tempted to mark a student present on the day of their 18th absence. (If student attendance rates are bunched right below the chronic absenteeism bar — say, many more are gone 17 rather than 18 days — that could be evidence of manipulation.)

“We need to use accountability to promote an early warning approach — not just making sure kids are one less day absent,” said Hedy Chang, the executive director of Attendance Works, a nonprofit that promotes efforts to improve school attendance.

Both Jordan and Schanzenbach noted that, because states are generally counting chronic absenteeism for only about 5 to 10 percent of school ratings, the incentives to cheat are likely to be fewer.

“It’s going be an empirical question about how big is the corruption of this — my prediction is it’s going to be reasonably small,” said Schanzenbach.

“This is why we encourage people to keep chronic absence to a relatively low percentage of the overall weighting — if it’s less than 10 percent … it’s not worth investing in trying to game it,” said Chang.

Still, attendance manipulation scandals have cropped up before: A 2016 investigation in Chicago, where student attendance rates are a part of school scores and principal evaluations, found that four high schools had systematically changed attendance records.

Others are concerned about data issues beyond obvious cheating.

“I’m worried about outright manipulation, but I’m also worried about sloppiness of reporting and inconsistencies,” said Chad Aldeman of Bellwether Education Partners, a reform-oriented consulting firm that has undertaken an extensive review of state ESSA plans. Details like how schools count partial-day absences, or what happens when a teacher forgets to take attendance, will take on new importance.

States are taking different approaches

Experts agrees that there should be some protections against manipulation of attendance data. But it’s unclear to what extent states have those safeguards in place.

Chalkbeat reached out to the 10 states that plan to use chronic absenteeism and have had their ESSA plans approved by the federal government. Nine of the state departments of education responded.

A representative for the Massachusetts Department of Education said that because the state can see changes to attendance rates, “any data manipulation would need to be done systematically and could not be done after the fact without raising flags as to why so many post-dated attendance changes were being made.”

Oregon already looks for unusual trends in attendance data, but does not conduct audits of local districts. “However, the accountability office has conducted such audits on other accountability data,” an education official said. “I would anticipate that we might do the same with attendance data submissions, should concerns arise regarding the validity of that data.”

But some states mention checks that might not catch most manipulation.

Illinois, for example, ensures that “a student can’t have more absences recorded than days enrolled in the school.”

A spokesperson from the Tennessee Department of Education noted just one kind of potential irregularity: “If the school said a student was in class all day but got in an accident or committed a crime during that window, the school could be liable.”

Some states said that they are working on this issue, but specifics have not yet been fleshed out.

Delaware “is reviewing current processes around absenteeism and chronic absenteeism.” Maine will be hiring an ESSA data coordinator “who will monitor data integrity,” but “the exact procedure and policy around this is in process of being written,” according to a spokesperson.

In Arizona, “the Department [of Education] is meeting with various stakeholders to determine ways to improve reporting of attendance and absenteeism.”

States that have already been using chronic absenteeism or attendance rates to determine funding may face less of a learning curve.

In its school performance reports, “New Jersey has included chronic absenteeism data for elementary and middle schools for years,” a spokesperson said.

Connecticut, meanwhile, has been “collecting and reporting chronic absenteeism data for many years” and has a number of checks in place, including flagging any school with large increases or decreases in chronic absenteeism, according to a spokesperson.

See all nine states’ full explanations for how they plan to protect against manipulation of absence data. Want more education news? Subscribe to Chalkbeat’s new national newsletter here.