Commissioner John King pledged this week to accept the “cut scores” recommended to him by a committee of educators, one of the final steps remaining before the state releases results from the state tests.
Cut scores determine the number of right answers students need on state English and math tests to be deemed proficient in the subjects.
The announcement at this month’s Board of Regents meeting came in the middle of a detailed 46-page slideshow presentation outlining how the “cut score” recommendations were made. But while the other slides were packed with numbers, graphs, and paragraphs, King’s 10-word acceptance of the standards got its own simple slide: “The Commissioner accepted recommendations from Day 5 with no changes.” (The full slideshow is below the jump.)
The flourish was a signal of the new transparency the department is trying to project around test scoring. In 2009, under then-Commissioner Richard Mills, dramatic improvements on state tests that had been seen as signs of academic progress across the state came under scrutiny for being inflated — not representing actual learning gains.
The inflation seems to have been the result of several factors, including focused test prep by teachers who became increasingly familiar with the tests. But at least one observer, Sol Stern, has reported that state officials might have deliberately inflated results by lowering cut scores so that more students would be deemed proficient. Commissioners do not have to accept the recommendations of the committee of educators that suggests where to set the scores.
And in his remarks yesterday, King told the Board of Regents that past commissioners have not always followed the recommendations that have come out of the scoring process. “Obviously, my view is that we should follow the recommendations of the [scoring process]. But it is worth noting that in the past in the department there have been departures from the recommendations,” he said.
King didn’t offer his judgment of those departures. But he did suggest that a commissioner’s decision not to follow the committee’s recommendation could be defensible — and even indicated that under certain circumstances, he might make that decision himself. “There have been times, and there may still be times, where the judged recommendations of the panel are not followed,” King said. “There can be a variety of public policy reasons why one wouldn’t follow them here.”
Asked about Stern’s report suggesting deliberate cut score inflation, a spokesman for the state education department said in an e-mail, “I have no information about what the Commissioner Mills did or did not do with his senior cabinet (none of whom still work here) or vendor partner (CTB McGraw Hill). Sol Stern’s speculation is just that, and we didn’t comment on it then and will not comment now.”
In a piece for City Journal, Stern reported on a phone conversation he had with Mills about the 2009 cut scores. “I asked Mills whether he’d had anything to do with setting the 2009 cut scores. He didn’t say either that he did or that he didn’t. ‘I did what was appropriate, and the record speaks for itself,’ he noted in our brief exchange,” Stern reported.
Attempts to reach Mills for comment yesterday were not successful.
This year, the state developed new and tougher assessments that are aligned to Common Core learning standards. It is widely expected that the passing rates will drop significantly again. Some estimate drops will be as large as 20 to 30 percentage points.
With the new tests on the way, officials dedicated this month’s Regents meeting to explaining the process known as “standard-setting,” a methodology used in the field of psychometrics to keep the bar of proficiency relatively even over time.
Kristen Huff, a senior fellow at the Regents Research Fund, led the presentation to the Board of Regents.
In late June, Huff explained, a panel of teachers, administrators and college professors convened in Troy for five days. For four days, 95 people broke out into smaller groups, studied the test items, and deliberated, like members of a jury, to determine the difficulty of each item. After several rounds of debate, they proposed cut scores that range from 100 to 425 points.
The cut scores set a minimum for students to reach one of the four proficiency levels that are now common to elementary and middle schools teachers, students, and parents in New York.
Ten percent of panelists said they would not defend the recommended cut scores, though none of them “strongly disagreed” with the recommendations.
A smaller group of 35 panelists stuck around for a fifth day to make minor adjustments to the recommended cut scores. The math mostly stayed the same, but the smaller panel made it slightly easier for students to score a Level 2 on the English exams for all grades, decreasing the minimum score needed by an average of two raw points.
Some who saw the presentation grumbled that the process was still not entirely transparent. Peter Goodman, an education blogger and teachers union delegate, said he wanted the department to disclose the names of the people picked to participate in the standard-setting.
Although the state’s methodology for choosing cut scores was reviewed by independent testing experts, Regent Kathy Cashin suggested the state should have gone further. She asked why the state had not appointed an independent expert to analyze the test items to asses the panel’s work. Assistant Commissioner Ken Wagner said that the role of the independent experts was only to examine the process.
Reliability of the tests is also determined in a technical report, which analyzes how students perform on certain test items. To date, the state has yet to release a report for the 2011-2012 tests and has not responded to questions asking to explain the delay. Pearson is contractually obligated to produce a draft of the reports within two weeks of the completion on the standard-setting.
The process will take longer this year because the tests are brand-new. That means that tests will come later than usual this year, Wagner said. Normally, they are released in mid-July, but officials said they don’t expect to release them until early August.