Liu: City hasn't gotten sufficient bang from ARIS's $83m buck

The Department of Education hasn’t gotten adequate bang for its buck from more than $80 million spent on ARIS, its data warehouse, concluded an audit released by Comptroller John Liu today.

Liu offered a solid clue to the audit’s conclusions last week, when he lambasted the city’s $10-million move to formally reassign its ARIS contract to Wireless Generation, which has managed the system for years.

The audit began in March 2011, shortly after Liu held a series of town hall meetings to solicit public input about what he should investigate. The data warehouse, launched in 2008 by IBM, has attracted no shortage of critics because of its steep price tag and early glitches.

Examining usage data, principals’ responses to a satisfaction survey the city administers, and the results of a survey that it distributed to educators in June, Liu’s office concluded principals’ satisfaction with ARIS has fallen, that many schools substitute other data programs in whole or in part, and that use among school staff has leveled off since the system’s first year, although use by department officials who work with schools has risen.

What’s more, the audit concludes, the city can’t show that ARIS is leading to higher student performance — something that former chancellor Joel Klein signaled would be a result when he rolled out the system in 2008.

“This costly tech program was much-touted by the DOE to help principals and teachers track progress and thereby improve student learning, even as long-time educators questioned its cost and effectiveness,” Liu said in a statement today. “$83 million later, there is little discernible improvement in learning and many principals and teachers have given up on the system.”

DOE officials disputed the audit’s methodology, conclusions, and very premise.

The audit assumes that the department believes ARIS alone can boost student achievement, writes Chief Academic Officer Shael Polakow-Suransky in the city’s formal response. In fact, he said, good teaching, not data, boosts achievement — and good teachers use data to help them.

Polakow-Suransky also took aim at the response rate of the survey Liu’s office administered. Of the 25,515 educators who received the survey, only 379 responded. That amounts to 1.5 percent of people surveyed and less than one half of one percent of the DOE employees who used it last year, Polakow-Suransky noted in his response.

“It is irresponsible of the Comptroller to use such a miniscule sample size to infer any significant conclusions; it is still more troubling that the Comptroller’s primary (only) basis for developing many of the Report’s conclusions is the results of such a survey,” he wrote.

Plus, Polakow-Suransky said, Liu’s office misrepresented survey results to present a skewed image of teachers’ satisfaction with ARIS. For example, he writes, only 14 percent of the people who responded to Liu’s survey said they found ARIS difficult to use. But the audit reports that 61 percent of respondents found ARIS “not very easy to use” — lumping in the 46.8 percent of respondents who picked the choice that said ARIS is “somewhat easy to use, but I would like to see changes made to it.”

The close of Liu’s 10-month audit does not signal the end of scrutiny on the ARIS system. Wireless Generation lost a $27 million contract to build a state system shortly after that company was acquired by Klein’s new outfit, the education unit of Rupert Murdoch’s News Corporation, then embroiled in a phone-hacking scandal. Now the state is on the hook to create that system or risk losing federal funding. Wireless Generation is also in the process of building a data system for several states, including New York, to share as part of the Gates Foundation’s Shared Learning Collaborative initiative.

The complete audit, which includes the city’s response, is below.