AP scores out for REACH NYC students — but what do they mean?

AP Tests by ##http://flickr.com/people/doctorow/##gruntzooki##

Last week, all four of the city’s major papers led their education news with previews of the first REACH NYC payout event, at which the privately funded consortium awarded city students cash prizes of up to $1,000 for scores on Advanced Placement exams. While some of the papers provided analysis of the numbers, all focused on the declining scores and suggested that the incentives program might actually have contributed to worsening outcomes for students. I want to revisit this coverage because, as with all education data, we need to ask ourselves what inferences we can legitimately make from the data we have available.

First, it’s important to note that last year’s “incentives” were announced well over a month into the fall semester, long after students had already started courses. And many of the schools selected for the pilot, including Frederick Douglass Academy in Harlem, already had strong AP track records. At least for its first year, we might better understand REACH as a “rewards” program for already-achieving students.

Still, we can make some inferences about the fact that the number of top scores increased, but the number of other passing scores decreased. This shift suggests that students who might already have taken the courses and the tests and done well continued to do so, but on the whole, new test takers — students who were already enrolled in AP classes but who, in the past, might have chosen not to take the exam — did less well. Generally speaking, as the number of students taking a particular test increases, the average score decreases, particularly when the original group was self-selecting. We saw this happen when the number of kids, particularly lower-income students, started taking the SAT in the early 1980s, and the same effect may have been at play this year in the nation’s declining ACT scores. It makes sense to expect that students who in the past have decided not to take an AP exam would, on average, score lower than their most motivated classmates.

It’s a lot harder to make inferences about the effect of REACH NYC’s incentives on students’ motivation. According to economist David Figlio on Eduwonkette last week, Columbia economist Jonah Rockoff has a working paper out that concludes that New York’s progress report accountability system rolled out last November influenced test scores this past spring. But because of the way AP tests are constructed, however, students’ scores are unlikely to prove as responsive to incentives or accountability systems as state test scores. First, AP tests are scored by a national panel of judges, unlike state tests, which are scored in house by teachers whose interest lies in their students’ success. In addition, scores are based on a test with multiple response types, not just multiple-choice questions. Simply put, AP exams are hard to game; the only reliable way to improve performance on them is to provide high-quality instruction to students who are already equipped with literacy, logic, and study skills. Cash incentives can’t make up for longstanding deficiencies in teaching and learning, no matter how motivated students might be to score high.

Of course, high scores might be an unnecessary objective. Jay Mathews of the Washington Post argues that simply exposing high school students to AP-level instruction, regardless of how they do on the exams, enriches their high school experience and makes them more likely to do well in college; he ranks high schools across the country by how many tests students attempt, not pass. By Mathews’ measure, we can predict that the AP incentives will this year benefit students, who knew about the possibility of a cash payoff at year’s end when they made their course selections (although even being able to elect AP courses means students have satisfied basic requirements that already make them likely to be able to succeed). But by the measure employed by REACH’s funders, who want to see low-income and minority students earn college credits by getting high scores on AP exams, students are likely to fall short no matter how much a high score can earn them.

The picture is also complicated by the fact that REACH NYC operates independently of Opportunity NYC, the citywide program that pays some students for grades, behavior, and attendance. Results from that program, which was designed by Harvard professor and the DOE’s Chief Equality Officer Roland Fryer, are due out this fall. But even before the results are made public, the program is being replicated in Washington, D.C. middle schools. As others rush to replicate New York’s experiments, it’s important that we identify just what we can fairly deduce from the information we have, about AP incentives and other programs. As Roland Fryer told the Times, “I don’t think we should make too much of the actual numbers at this point.”