“Welcome to the dark side of Big Data,” growls the last line of the first chapter of Cathy O’Neil’s recent book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. As that sentence (and that subtitle) suggest, this is not a subtle book. O’Neil chronicles harms from the widespread use of machine learning and other big data systems in our society. O’Neil is convinced that something ominous and harmful is afoot, and she lays out a bill of particulars listing dozens of convincing examples.
This is a book that I like (lots) because we need outspoken and authoritative chroniclers of the downsides of big data decisionmaking. It advances a carefully articulated and well-supported argument, delivered with urgency and passion. For readers yearning for a balanced look at both the benefits and the costs of our increasingly automated society, however, keep searching.
If we built a prototype for a qualified critic of big data, her background would look a lot like O’Neil’s: Harvard math PhD, MIT postdoc, Barnard professor, hedge fund quant during the financial crisis, start-up data scientist. Throw in blogger (mathbabe.org) and Occupy organizer for good measure, and you cannot quibble with the credentials. O’Neil is an author who knows what she is talking about, who also happens to be a writer of compelling, clear prose, an evidently skilled interviewer, and a great speaker.
Perhaps most importantly, the book provides legal scholars with a concise and salient label—weapons of math destruction, or WMDs—to describe decisionmaking algorithms possessing three features: opacity, scale, and harm. This label and three-factor test can help us identify and call out particularly worrisome forms of automated decisionmaking.
For example, she seems to worry most—and have the most to say—about so-called “value added modeling” systems for assessing the effectiveness of teachers in public schools. Reformers such as Michelle Rhee, former Chancellor of the DC public schools, spurred by policies such as No Child Left Behind, embraced a data-centric model, which selected which teachers to fire based heavily on the test scores of their students. The affected teachers had little visibility into the magic formulae that decided their fate (opacity); these tests affected thousands of teachers around the country (scale); and good teachers were released from important jobs they loved, depriving their students of their talents (harm). When opacity, scale, and harm align in an algorithmic decisionmaking system, software can worsen inequality and ruin lives.
Building on these factors, O’Neil returns repeatedly to the important role of feedback in exacerbating (and sometimes blunting) the harm of WMDs. If we use the test results of students to identify topics they are not learning, to change what or how we are teaching, this is a positive and virtuous feedback loop, not a WMD. But when we decide to fire the bottom five percent of teachers based on those same scores, we are assuming the validity and accuracy of the test, making it impossible to use feedback to test the strength of those assumptions. The critical role of feedback is an important key insight of the book.
The book brims with other examples of WMDs, devoting considerable attention to criminal recidivism scoring systems, employment screening programs, predictive policing algorithms, and even the U.S. News college ranking formula. O’Neil spends entire chapters covering big data systems that stand in our way of getting a job, succeeding at work, buying insurance, and securing credit.
Legal scholars who write about automated decisionmaking or artificial intelligence may be surprised to see this book reviewed in these pages. O’Neil’s book is long on description with very little attention paid to policy solutions. A book of deep legal scholarship, this is not. As capably as she writes about math and algorithms, O’Neil falters—and I’m guessing she would cop to this—when it comes to law and regulation, mixing equal parts unrealistically optimistic sentiments about laws like FCRA; vague descriptions about the prospect of Constitutional challenges to data practices; and unrealistic calls for new legislation.
Despite these extra-disciplinary shortcomings, this book should be read by legal scholars, who are not likely to already know all the stories in this book and who will find many compelling (if chilling) examples to cite. As one who does not focus on education policy, for example, I was struck by the detailed and personal stories of teachers fired because of the whims of value-added modeling. And even for the old stories I had heard before, I was struck by how well O’Neil tells them, distilling complicated mathematical concepts into easy-to-digest descriptions and using metaphor and analogy with great skill. I will never again think of a model without thinking of O’Neil’s lovely example of the model she uses to select what to cook for dinner for her children.
The book is in parts intemperate. But we live in intemperate times, and the problems with big data call for an intemperate call-to-arms. A more measured book, one which tried to mete out praise and criticism for big data in equal measure, would not have served the same purpose. This book is a counterpoint to the ceaseless big data triumphalism trumpeted by powerful partisans, from Google to the NSA to the U.S. Chamber, who view unfettered and unexamined algorithmic decisionmaking as their entitlement and who view criticism of big data’s promise as an existential threat. It responds as well to big data’s academic cheerleaders, who spread the word about the unalloyed wonderful potential for big data to drive innovation, grow the economy, and save the world. A milquetoast response would have been drowned out by these cheery tales, or worse, co-opted by them.
“See,” big data’s apologists would have exclaimed, “even Cathy O’Neil agrees about big data’s important benefits.” O’Neil is too smart to have written a book that could have been co-opted in this way. “Big Data has plenty of evangelists, but I’m not one of them,” O’Neil proudly proclaims. Neither am I, and I’m glad that we have a thinker and writer like O’Neil shining a light on some of the worst examples of the technological futures we are building.