How and why does it matter that humans do things that machines might do instead, more quickly, consistently, productively, or economically? When and where should we care that robots might take our jobs, and what, if anything, might we do about that?
It is the law’s turn, and the law’s time, to face these questions. Richard Re and Alicia Solow-Niederman offer an excellent, pragmatic overview and framework for thinking about artificial intelligence (AI) in the courtroom. What if the judge is a (ro)bot?
The general questions are far from novel, and there is no shortage of recent research. Facing the emergence of computable governance in the workplace and large swaths of social life, for the last twenty years legal scholars, historians, and researchers in science and technology studies have been exploring “algorithmic” decision-making in computer networks, social media platforms, the “gig” economy, and wage labor.
Yet the application of automation to the law feels different, disconcerting, and disruptive for added reasons that are not always easy to identify. Is it the central role that law and legal systems play in constructions of well-ordered modern society? Is it the combination of cognition and affect that define modern lawyering and judging? Is it the narrative of law as craft work that still underpins so much legal education and practice? Those questions form the conceptual underpinnings of Re and Solow-Niederman’s work. They are after a framework for organizing thoughts about the answers, rather than the answers themselves.
Organizing a framework means pragmatism rather than high theory. The problem to be addressed is “how can we distinguish the benefits from the harms associated with automated judging?” rather than “What defines the humanity of the law?” Re and Solow-Niederman address courtroom practice and judicial decision-making as their central example.
The article proceeds elegantly in a handful of steps.
First, Re and Solow-Niederman propose a reconfigured model of systems-level interactions between “law” and “technology,” shifting from the law as a set of institutions that “responds” to technological innovation (a linear model, labelled “Rule Updating”) and toward law as a set of institutions whose capacities co-evolve with technological innovation (a feedback-driven model, labelled “Value Updating”).
Within the Value Updating model, the article addresses adjudication, distinguishing between stylized “equitable justice” and stylized “codified justice.” The former is usually associated with individualized proceedings in which judges apply legal rules and standards within recognized discretionary boundaries. The latter is usually associated with the routinized application of standardized procedures to a set of facts. The justice achieved by a system of adjudication represents a blend of interests in making accurate decisions and making just decisions.
Re’s and Solow-Niederman’s concerns arise with the alignment and reinforcement of codified justice by algorithmic systems, the “artificially intelligent justice” of their title. They acknowledge that what they call codified justice is not new; they invoke precedents in the federal sentencing guidelines and matrices for administering disability benefits. Nor is codified justice, in its emerging AI-supported forms, temporary. Algorithmic judging supported by machine learning is here to stay, particularly in certain parts of criminal justice (for example, parole and sentencing determinations) and benefits administration, and its role is likely to expand.
Re and Solow-Niederman argue that the emergence of AI in adjudication may shift existing balances between equitable justice and codified justice in specific settings, in ways that key into macro shifts in the character of the law and justice. Their Value Updating model renders those shifts explicit. With AI-based adjudication, they argue that we may see more codified justice and less equitable justice. Why? Because, they note, motivations for adoption and application of AI to adjudication are tangible. Codified justice promises to be relatively cheap; equitable justice is relatively expensive. Firms are likely to promise and to persuade, rightly or wrongly, that AI may deliver better, faster, and cheaper decision-making at scale.
The article is careful to note that these shifts are not inevitable but that the risks and associated concerns are real. Perhaps the most fundamental of those concerns is that AI-supported changes to adjudication may shift “both the content of the law and the relationship between experts, laypersons, and the legal system in democratic society” (P. 262) in systematic ways. Decision-making and adjudicative outcomes may be incomprehensible to humans. Data-driven adjudication may limit the production or persuasiveness of certain types of system-level critiques of legal systems, and it may limit the extent to which rules themselves are permitted to evolve. Reducing the role of human judges may lead to system-level demoralization and disillusionment in society as a whole, leading to questions of legitimacy and trust not only with respect to adjudicative systems but regarding the very architecture of democracy. To paraphrase Re’s and Solow-Niederman’s summation: if robots resolve disputes, why should humans bother engaging with civil society, including fundamental concepts of justice and the identity and role of the state?
Re and Solow-Niederman conclude with their most important and most pragmatic contributions, describing a range of stylized responses to AI’s promise of “perfect enforcement of formal rules” (P. 278) that illuminate “a new appreciation of imperfect enforcement.” (Id.) Existing institutions and systems might be trusted to muddle through, at least for a while, experimenting with AI-based adjudication in various ways without committing decisively to any one approach. Alternatively, equitable adjudication could be “coded into” algorithmic adjudicators, at least in some contexts or with respect to some issues. A third approach would involve some systematic allocation of adjudicative roles to humans rather than machines, a division of labor approach. A final response would tackle the problems of privately developed robot judges by competing with them, via publicly supported or endorsed systems. If you can’t join them (or don’t want to), beat them, as it were. As with the article as a whole, this survey of options is inspired by broad conceptual topics, but its execution has an importantly pragmatic character.
Little of the material is fully novel. The work echoes themes raised several years ago by Ian Kerr and Carissima Mathen and extended more recently by Rebecca Crootof, among others. Its elegance lies in the coordination of prior thinking and writing in an unusually clear way. The framework can be applied generally to the roles that algorithms increasingly play in governance of many sorts, from urban planning to professional sports.
I’ll close with an illustration of that point, one that appears mostly, and briefly, in the footnotes. Consider soccer, or football, as it is known in much of the world. Re and Solow-Niederman acknowledge the utility of thinking about sports as a case study with respect to automation and adjudication. (P. 254 n. 37; P. 278 n. 121.) The following picks up on this and extends it, to show how their framework can be applied to help clarify thinking about a specific example. Other scholars have done similar work, notably Meg Jones and Karen Levy in Sporting Chances: Robot Referees and the Automation of Enforcement. But they did not include soccer in their group of cases, and automation in soccer refereeing has some distinctive attributes that may be particularly relevant here.
A few years ago, to improve refereeing in professional football matches, VAR systems (short for Video Assistant Referee) were introduced. During breaks in play, referees are permitted to look at recorded video of the game and consult with off-field officials who supervise video playback.
VAR has been controversial. It has been implemented so far in a “division of labor” sense, against a long history of experimentation with the rules of the game (or “laws,” as they are formally known). VAR data are generally determinative with respect to rule-based judgments, such as whether a goal has been scored. VAR data are employed differently with respect to possible penalty kicks and possible ejections. In both contexts, presumably because of the severity of the consequences (or, perhaps, despite them), VAR data are advisory. The human referee retains the discretion to make final determinations.
The relevance of VAR is not its technical details; the point is its systems impact. As Jones and Levy note, a mechanical element has been introduced in a game in which both play and adjudication have long been inescapably error-prone. Rightness and wrongness, even in a yes/no sense, are human and humane constructs, in soccer and in the law. VAR, like an AI judge, changes something about this human “essence” of playing and judging experiences.
But the VAR example illuminates something critical about Re and Solow-Neiderman’s framework. Soccer referees not only adjudicate yes/no applications of the rules. Penalty kicks and player ejections do not follow only from administration of soccer’s laws in a “correct/incorrect” sense, with accuracy as the paramount value. In the long history and narrative of soccer, the referee’s discretion has always represented justice. Does a violent tackle warrant a penalty kick? Sometimes it does; sometimes it does not. Unlike referees’ decisions in other sports with machine-based officiating, critical judgments in soccer are based on “fairness” rather than only on “the rule,” where “fairness” is equated to a sense of earned outcomes, or “just deserts.” The soccer referee is dispenser of what might be called “equitable justice” on the field. Enlisting VAR risks tilting this decision-making process toward what might be called “codified justice.”
Is this good for the game, or for the society that depends on it? It’s too soon to say. Soccer, like all institutions, has never been unchanging. Soccer laws, soccer technologies, and soccer values are always at least a little bit in flux, and sometimes much more so. But the soccer example offers not simply another way of understanding challenges of AI and the law. Re and Solow-Neiderman have given us a framework based in the law that helps us understand the challenges of automation and algorithms across additional critical domains of social life. Those challenges ask us to consider, again, what we mean by justice—not only in the law but also beyond it.
Cite as: Michael Madison, Oyez! Robot
(January 24, 2020) (reviewing Richard M. Re & Alicia Solow-Niederman, Developing Artificially Intelligent Justice
, 22 Stan. Tech. L. Rev
. 242 (2019)), https://cyber.jotwell.com/oyez-robot/
Kevin Werbach, The Blockchain and the New Architecture of Trust (2018).
The distributed ledger technology known as the blockchain has continued to gather interest in legal academia. With a growing number of books and academic papers exploring almost every aspect of the subject, it is always good to have a comprehensive book that not only covers the basics, but also provides new insights. Kevin Werbach’s The Blockchain and the New Architecture of Trust provides an in-depth yet easy-to-comprehend analysis of the most important aspects of blockchain technology. It also manages to convey astute analysis and some needed and sobering skepticism about the subject.
Werbach describes the characteristics of blockchains, providing a thorough and easy-to-understand introduction to key concepts and the reasons why the technology is thought to be a viable solution for various problems. A blockchain is a cryptographic distributed ledger that appends information in a supposedly immutable manner. An open record of all transactions is made public, which then communicates a “shared truth”, that is a single proof that everyone trusts because it has been independently verified by a majority of the participants in the network. This technology is thought to be a solution for problems such as the lack of reliability in accounting records, and for double spending, where the same amount of currency is used in two different transactions. While the first half of the work is likely to be the most useful for those who are not familiar with the underlying technical concepts, the book really makes its best contributions in its later chapters, where the author begins to criticize various aspects of the technology.
A core topic of the book is that of “a crisis of trust,” and Werbach explains how the blockchain has been repeatedly offered as a possible solution to this problem. Everyday life is built on trust, and we are always engaged in rational calculations based on trust-related questions: Can I trust this person with my car keys? Can I trust my bank? Can a lender trust a borrower? We have built systems of risk assessment, of managing trust, and of redress when trust breaks down. While our financial and legal systems rely on a good measure of trust, it is seen by many as an added expense and an obstacle to a well-functioning economy. Werbach explains how in some circles the concept of trust has become “almost an obscenity.” The need for trust is a vulnerability, a flaw that should be remedied.
Enter the blockchain, a distributed ledger technology that is supposed to solve this perceived crisis of trust by being “trustless”—that is, doing away with the need for trust. The blockchain is proposed as a better solution than past architectures of trust, such as centralized “Leviathans,” third-party arbiters, or peer-to-peer systems. States or other large central authorities historically addressed the problem of trust through enforcement. Alternatively or additionally, intermediaries have acted as arbiters and providers of trust. In peer-to-peer systems, trust is built on an assumption of shared values and norms, and on principles of self-governance.
Werbach explains why the blockchain is characterized as better:
In any transaction, there are three elements that may be trusted: the counterparty, the intermediary, and the dispute resolution mechanism. The blockchain tries to replace all three with software code. People are represented through arbitrary digital keys, which eliminates the contextual factors that humans use to evaluate trustworthiness. (P. 29.)
Werbach identifies a problem, however: for a system that appears to rely so much on the concept of trustlessness, there is still a lot of trust required in intermediaries in the blockchain space. This observation proves Nick Szabo’s comment that “there is no such thing as a fully trustless institution or technology”. Very few people have the technical know-how to become fully self-reliant in the use of blockchain technology, so most people end up having to trust some form of intermediary. The book covers several disasters, such as the infamous QuadrigaCX fiasco, where users lost millions due to either negligence or fraud. “The blockchain was developed in response to trust failures, but it can also cause failures.” (P. 117.)
Werbach cites other examples where intermediaries have exercised a large amount of control, such as the Decentralized Authority Organization (DAO) hack. The DAO was a way to manage transactions on the Ethereum blockchain using smart contracts and a shared pool of funds. A hacker found a way to syphon out funds through a bug in the code. A truly trustless decentralized system would allow this to happen, but the DAO developers decided to fork the code, re-writing history, erasing the bug, and creating a new version of the blockchain where the hack never occurred. This has been seen as evidence that trust is still involved—a trust in blockchain developers and intermediaries.
Werbach also identifies that the blockchain failed in practice to be a fully decentralized system. Satoshi Nakamoto, the pseudonymous inventor of the concept of the blockchain, dreamt of a system where millions of individual miners would come together to verify transactions, making the blockchain a truly decentralized endeavor. But what has happened is more centralized, with mining concentrated in a few massive conglomerates.
Werbach makes some very good criticisms of another use of the blockchain, namely smart contracts. There are various reasons for distrusting smart contracts, but one of the most compelling offered in the book is that despite advances in machine learning, “computers do not have the degree of contextual, domain specific knowledge or subtle understanding required to resolve contractual ambiguity.” (P. 125.)
Finally, Werbach goes into several interesting discussions specific to some platforms, and looks at regulatory responses. I found the regulatory discussion interesting, but perhaps the least useful aspect of the book. The attempts to regulate the blockchain phenomenon move so fast that a few of the examples offered are already outdated, even though the book was published in 2018. As a European reader, I would also have liked a bit more coverage of international developments; while the author cites several cases from abroad, these were not dealt with in depth.
However, these are minor concerns. This is a thorough, informative, and highly readable book that should be the go-to reference for anyone interested in the subject of blockchain and the law.
For all the justifiable concern in recent years directed toward the prospect of autonomous weapons, other military uses of automation may be more imminent and more widespread. In Predicting Enemies, Ashley Deeks highlights how the U.S. military may deploy algorithms in armed conflicts to determine who should be detained and for how long, and who may be targeted. Part of the reason Deeks predicts these near-term uses of algorithms is that the military has models: algorithms and machine-learning applications currently used in the domestic criminal justice and policing contexts. The idea of such algorithms being employed as blueprints may cause heartburn. Their use domestically has prompted multiple lines of critique about, for example, biases in data and lack of transparency. Deeks recognizes those concerns and even intensifies them. She argues that concerns about the use of algorithms are exacerbated in the military context because of the “double black box”—“an ‘algorithmic black box’ inside what many in the public conceive of as the ‘operational black box’ of the military” (P. 1537)—that hampers oversight.
Predicting Enemies makes an important contribution by combining the identification of likely military uses of algorithms with trenchant critiques drawn from the same sphere as the algorithmic models themselves. Deeks is persuasive in her arguments about the problems associated with military deployment of algorithms, but she doesn’t rest there. She argues that the U.S. military should learn from the blowback it suffered after trying to maintain secrecy over post-9/11 operations, and instead pursue “strategic transparency” about its use of algorithms. (P. 1587.) Strategic transparency, as she envisions it, is an important and achievable step, though likely still insufficient to remedy all of the concerns with military deployment of algorithms.
Deeks highlights several kinds of algorithms used domestically and explains how they might parallel military applications. Domestic decision-makers use algorithms to assess risks individuals pose in order to determine, for example, whether to grant bail, impose a prison sentence, or allow release on parole. Even more controversially, police departments use algorithms to “identif[y] people who are most likely to be party to a violent incident” in the future (P. 1543, emphasis omitted), as well as to pinpoint geographic locations where crimes are likely to occur.
These functions have military counterparts. During armed conflicts, militaries often detain individuals and have to make periodic assessments about whether to continue to detain them based on whether they continue to pose a threat or are likely to return to the fight. Militaries, like police departments, also seek to allocate their resources efficiently. Algorithms that predict where enemy forces will attack or who is likely to do the attacking, especially in armed conflicts with non-state armed groups, would have obvious utility.
But, Deeks argues, problems with domestic use of algorithms are exacerbated in the military context. As compared with domestic police departments or judicial officials, militaries using algorithms early in a particular conflict are likely to have far less and less granular information about the population with which to train their algorithms. And algorithms trained for one conflict may not be transferable to different conflicts in different locations involving different populations, meaning that the same problems with lack of data would recur at the start of each new conflict. There’s also the problem of applying algorithms “cross-culturally” in the military context, rather than “within a single society” as is the case when they are used domestically (P. 1565), and the related possibility of exacerbating biases embedded in the data. With bad or insufficient data come inaccurate algorithmic outcomes.
Deeks also worries about “automation bias”—that military officials will be overly willing to trust algorithmic outcomes and even more susceptible to this risk than judges, who are generally less tech-savvy. (Pp. 1574-75.) At the same time, she also warns that a lack of transparency about how algorithms work could make military officials unwilling to trust algorithms when they should, that is, when the algorithms would actually improve decision-making and compliance with international law principles like distinction and proportionality. (Pp. 1568-71.)
These and other concerns lead Deeks to her prescription for “strategic transparency.” Deeks argues that the military should “fight its institutional instincts” (P. 1576) to hide behind classification and limited oversight from Congress and the public and instead deploy a lesson from the war on terror—that “there are advantages to be gained by publicly confronting the fact that new tools pose difficult challenges and tradeoffs, by giving reasons for their use, and by clarifying how the tools are used, by whom, and pursuant to what legal rules.” (P. 1583.) Specifically, Deeks argues that in pursuing transparency, the military should explain when and how it uses algorithms and machine learning, articulate how such tools comply with its international law obligations, and engage in a public discussion of costs and benefits of using algorithms. (Pp. 1588-89.) She also urges the military to “articulate how it will test the quality of its data, avoid training its algorithms on biased data, and train military users to avoid falling prey to undue automation biases.” (P. 1590.)
Deeks previously served as the Assistant Legal Adviser for Political-Military Affairs in the State Department’s Office of the Legal Adviser (where I had the pleasure of working with her), and so she has the experience of an internal advisor combined with the critical eye of an academic commentator. One hopes that the U.S. military—and others around the world—will heed her thoughtful advice about transparency in the use of algorithms. Transparency is not a panacea for problems of data availability, quality, and bias, but it may help with oversight and accountability. And that’s a good first step.
Software crashes all the time, and the law does little about it. But as Bryan H. Choi notes in Crashworthy Code, “anticipation has been building that the rules for cyber-physical liability will be different.” (P. 43.) It is one thing for your laptop to eat the latest version of your article, and another for your self-driving lawn mower to run over your foot. The former might not trigger losses of the kind tort law cares about, but the latter seems pretty indistinguishable from physical accidents of yore. Whatever one may think of CDA 230 now, the bargain struck in this country to protect innovation and expression on the internet is by no means the right one for addressing physical harms. Robots may be special, but so are people’s limbs.
In this article, Choi joins the fray of scholars debating what comes next for tort law in the age of embodied software: robots, the internet of things, and self-driving cars. Meticulously researched, legally sharp, and truly interdisciplinary, Crashworthy Code offers a thoughtful way out of the impasse tort law currently faces. While arguing that software is exceptional not in the harms that it causes but in the way that it crashes, Choi refuses to revert to the tropes of libertarianism or protectionism. We can have risk mitigation without killing off innovation, he argues. Tort, it turns out, has done this sort of thing before.
Choi dedicates Part I of the article to the Goldilocksean voices in the current debate. One camp, which Choi labels consumer protectionism, argues that with human drivers out of the loop, companies should pay the cost of accidents caused by autonomous software. Companies are the “least cost avoiders” and the “best risk spreaders.” This argument tends to result in calls for strict liability or no-fault insurance, neither of which Choi believes to be practicable.
Swinging from too hot to too cold, what Choi calls technology protectionism “starts from the opposite premise that it is cyber-physical manufacturers who need safeguarding.” (P. 58.) This camp argues that burdensome liability will prevent valuable innovation. This article is worth reading for the literature review here alone. Choi briskly summarizes numerous calls for immunity from liability, often paired with some version of administrative oversight.
Where Goldilocks’s bears found happiness in the third option, Choi’s third path forward is found wanting. What he calls doctrinal conventionalism takes the view that tort law as-is can handle things. Between negligence and strict products liability, this group argues, tort will figure robots out.
This third way, too, initially seems unsatisfactory. The law may be able to handle technological developments, Choi acknowledges, but the interesting question isn’t whether; it’s how. And crashing code, he argues in Part II, is at least somewhat exceptional. Its uniqueness isn’t in the usual lack of physical injuries, or the need for a safe harbor for innovation. It’s the problem of software complexity that makes ordinary tort frameworks ill-suited for governing code. Choi explains that software complexity makes it impossible for programmers to guarantee a crash-free program. This “very basic property of software…[thus] defies conventional judicial methods of assessing reasonableness.” (P. 79.) No matter how many resources one applied to quality assurance, one could “still emerge with errors so basic a jury would be appalled.” (P. 80.) (Another bonus for the curious: Choi’s discussion of top-down attempts to bypass these problems through mandating particular formal languages in high-stakes fields such as national security and aviation.)
The puzzle, then, isn’t that software now produces physical injuries, thus threatening the existing policy balance between protecting innovation and remediating harm. It’s that these newly physical injuries make visible a characteristic of software that makes it particularly hard to regulate ex post, through lawsuits. In other words, “[s]oftware liability is stuck on crash prevention,” when it should be focused instead on making programmers mitigate risk. (P. 87.)
In Part III, Choi turns to a line of cases in which courts found a way to get industry to increase its efforts at prevention and risk mitigation, without crushing innovation or otherwise shutting companies down. In a series of crashworthiness cases from the 1960s, courts found that car manufacturers were responsible for mitigating injuries in a car crash, even if (a) such crashes were statistically inevitable, and (b) the chain of causation was extremely hard to determine. While an automaker might not be responsible for the crash itself, it could be held liable for failing to make crashing safer.
Crashworthiness doctrine, Choi argues, should be extended from its limited use in the context of vehicular accidents to code. In the software context, he argues that “there are analogous opportunities for cyber-physical manufacturers to use safer designs that can mitigate the effects of a software error between the onset and the end of a code crash event.” (P. 101.) Programmers should be required not to prevent crashes entirely, but to use known methods of fault tolerance, which Choi discusses in detail. Courts applying crashworthiness doctrine to failed software thus would inspect the code to determine whether it used reasonable fault tolerance techniques.
Crashworthy Code could be three articles: one categorizing policy tropes; one identifying what makes software legally exceptional or disruptive; and one discussing tort law’s ability to handle risk mitigation. But what is most delightful about it is Choi’s thoroughness, his refusal to simplify or overreach. There is something truly delicious about finding a solution to a “new” problem in a strain of case law from the 1960s that most people probably don’t know existed. This is what common law is good at: analogies. And this is what the best technology lawyers among us are best at: finding those analogies, and explaining in detail why they fit technology facts.
Ian Kerr 1965-2019
Ian Kerr, who passed away far too young in 2019, was an incisive scholar and a much treasured colleague. The wit that sparkled in his papers was matched only by his warmth toward his friends, of whom there were many. He and his many co-authors wrote with deep insight and an equally deep humanity about copyright, artificial intelligence, privacy, torts, and much much more.
Ian was also a valued contributor to the Jotwell Technology Law section. His reviews here display the same playful generosity that characterized everything else he did. In tribute to his memory, we are publishing a memorial symposium in his honor. This symposium consists of short reviews of a selection of Ian’s scholarship, written by a range of scholars who are grateful for his many contributions, both on and off the page.
Ellen P. Goodman & Julia Powles, Urbanism Under Google: Lessons from Sidewalk Toronto,
__ Fordham L. Rev.
__ (forthcoming 2019), available at SSRN
National Geographic’s April 2019 issue focused on ‘cities’, presenting photographs, highlighting challenges, and wondering about the future. Its editor highlighted that two-thirds of the world’s population is expected to live in a city by 2050, and recent history is replete with unfinished or abandoned blueprints for what this future might look like. Yet in the field of technology law and urban planning, the biggest story of the last two years may well be that of Toronto, where a proposal to rethink urban life through data, technology, and redevelopment has prompted important reflections on governance, privacy, and control.
In Urbanism Under Google: Lessons from Sidewalk Toronto, forthcoming in the Fordham Law Review, Ellen P. Goodman and Julia Powles set out to tell the story of the ‘Sidewalk Toronto’ project, from its early announcements (full of promise but lacking in detail) to the elaborate (yet no less controversial) legal and planning documents now publicly available. Goodman and Powles contribute to the public and academic scrutiny of this specific project, but their critique of process and transparency will obviously be of value in many other cities, especially as ‘smart city’ initiatives continue to proliferate.
Sidewalk Labs, associated with Google through its status as a subsidiary of Alphabet (Google’s post-restructuring parent company), is working with Waterfront Toronto (the tripartite agency consisting of federal, provincial and municipal government) to redevelop a soggy piece of waterfront land, ‘Quayside’. Or is it? One of Goodman and Powles’ main observations, splendidly delivered as an a-ha moment halfway through the piece, is how the relationship between, on one hand, the Quayside proposal and, on the other, the wider idea of redeveloping waterfront lands has come to public attention. And indeed, the complexity of the project and its associated documents must have been a key driver for Goodman and Powles, as much of the article’s contribution comes from its careful and close reading of the extensive documentation now published by Sidewalk and by Waterfront Toronto.
Sidewalk Toronto has prompted many reactions. Some are enthusiastic about the promise of a great big beautiful tomorrow. Others see a dystopian surveillance (city-)state where every move is not just tracked but monetised. In Goodman and Powles’ account, the focus is on two issues: (1) the corporate and democratic issues arising out of the relationship between Waterfront Toronto and Sidewalk Labs (including the role of the different levels of government and scrutiny operating in Canada) (part 2), and (2) the specific ‘platform’ vision set out by Sidewalk in its plans (part 3).
The corporate and democratic issues highlighted primarily relate to the role of Waterfront Toronto, which is itself a distinctive agency within the Canadian administrative state. The authors argue that insufficient detail about the project was provided at early stages, and that the processes for public engagement did not allow for sufficient participation by the public (especially as a result of information asymmetries). Various incidents from this period, including leaks, resignations, and media commentary, are set out.
Considering the ‘platform’ vision, a set of concerns trouble the authors. Drawing upon Sidewalk’s own textual and visual representations of its intentions, they identify efficiency and datafication as the key outcomes of a platform-led approach, and situate these developments in a wider ‘smart city’ literature. Testing Ben Green’s work on asymmetries of power, and applying broader concepts such as legal scholar Brett Frischmann’s account of infrastructure and geographer Rob Kitchin’s discussion of the embedding of values, they find that Sidewalk’s approach is radical (or at least an exaggerated version of what is already seen in some smart city initiatives) and potentially a concentration of great power and influence.
One of the most fascinating concepts to emerge from Sidewalk’s plans, and the vibrant debate that these plans have provoked in Canada and elsewhere, is ‘urban data’. Sidewalk defines this broadly as data collected in public spaces and certain other spaces, normally de-identified. Goodman and Powles respond to this emergent (and non-statutory) category in both parts 2 and 3. They are not convinced by de-identification, self-certification, or the novelty of what Sidewalk proposes. Nor are they reassured by the emergent (and fashionable) idea of ‘data trusts’ as a way of addressing legal and popular concerns about the impact of initiatives like this on privacy and intellectual property, and questions of ownership and control associated with both issues.
On their return to the urban data’ issue in the later pages, and bringing together the democratic and platform issues, the authors raise a set of broader questions about control over shared spaces (adding to Lilian Edwards’ influential exploration of public/private divides in smart cities). Neatly, then, Lawrence Lessig’s rhetorical and conceptual use of the history of road building and city planning as paving the way for an understanding of (information) architecture as law comes full circle, as Goodman and Powles wonder how control over a digital layer calls into question the norms of planning and land use.
The authors end on a somewhat resigned note, highlighting the infamy and ‘inevitability’ of the project. Today, though, there is still some doubt as to whether Sidewalk’s plans will come to fruition in their current form. As the Toronto-based scholar Natasha Tusikov, well known to scholars of technology law through her pioneering investigation of industry-led governance of Internet ‘chokepoints’, has just written, the June 2019 release of Sidewalk Toronto’s master plan has led into a new and much-watched round of consultations. Sidewalk’s next steps may therefore disclose the extent to which the critiques set out here by Goodman and Powles can, if at all, be resolved within this project—although their analysis can also inform the work that other cities and development agencies might be planning.
Cite as: Daithí Mac Síthigh, We the North
(September 3, 2019) (reviewing Ellen P. Goodman & Julia Powles, Urbanism Under Google: Lessons from Sidewalk Toronto,
__ Fordham L. Rev.
__ (forthcoming 2019), available at SSRN), https://cyber.jotwell.com/we-the-north/
Ian Kerr 1965–2019
Our community, and the Jotwell Tech section, lost a giant this week. Ian Kerr, who died on August 27 of complications from cancer, was a contributing editor to this section since its founding in 2009, and a luminary of the field of law and technology for much longer. As Canada Research Chair in Law, Ethics, and Technology at the University of Ottawa, Kerr was a leader in in taking ethics and interdisciplinarity seriously. His expansive body of work often addressed questions years before others saw them.
Kerr embodied the Jotwell mission statement: to “identify, celebrate, and discuss the best.” He was unafraid to be laudatory; he was positive without apology. His writing could be bitingly funny, and he loved to let the air out of overinflated ideas, but he was unfailingly, irrepressibly generous toward his fellow scholars and his fellow humans. He won teaching award after teaching award. He welcomed, mentored, and was a constant source of encouragement and critical feedback for everyone—senior or junior, professor ,or student.
In Kerr’s words (about his alums and research team), we are “incredibly fortunate to find [ourselves] surrounded by such excellent people.” He himself was one of these excellent people, not least because he brought out what was excellent in everyone around him. We will miss his voice, his leadership, his brilliance, his friendship, his kindness and his irreverent grace.
Tarleton Gillespie’s important book Custodians of the Internet unpacks the simultaneous impossibility and necessity of content moderation, highlighting nuance rather than answering questions. Within big companies, content moderation is treated like custodial work, like sweeping the floors—and recent revelations reinforce that the abjectness of this work seems to contaminate those who do it. The rules are made by people in positions of relative power, while their enforcement is traumatic, poorly-paid, outsourced scutwork. But for major platforms, taking out the trash—making sure the site isn’t a cesspool—is in fact their central function.
Gillespie urges us to pay attention to the differences between a content policy—which is a document that both tries to shape the reactions of various stakeholders and is shaped by them—and actual content moderation; both are vitally important. (Facebook’s newly announced “Supreme Court” is on the former side: it will make important decisions, but make them at a level of generality that will leave much day-to-day work to be done by the custodial staff.) Every provision of a content policy represents a horror story and also something that will definitely be repeated. Gillespie is heartbreakingly clear on the banality of evil at scale: “a moderator looking at hundreds of pieces of Facebook content every hour needs more specific instructions on what exactly counts as ‘sexual violence,’ so these documents provide examples like ‘To snap a bitch’s neck, make sure to apply all your pressure to the middle of her throat’—which Facebook gives a green checkmark, meaning posts like that should stay.” (P. 112.)
Given the scale of decision-making, what else could be done? The Apple App Store preapproves apps, but constantly struggles because of the need for speed and the inevitable cries of censorship when Apple decides an app is too political. And Apple can get away with preapproval only because the flood of apps is orders of magnitude less than the flood of Instagram photos or similar content, and because app developers are relatively more likely to share ideas about reasonable expectations of a platform than are web users in general.
Most other platforms have relied on community flagging in the first instance. Relying on the users to report the need for clean-up is practically convenient, but also grants “legitimacy and cover” by signaling that the platform is listening to and trying to help users who are being harmed, while still leaving ultimate control in the platform’s hands. Mechanisms to report content can also be used by harassers, such as the person who reported hundreds of drag queens for violating Facebook’s real-name policy. YouTube sees flags surge when a new country gets access to the platform; Gillespie’s informant attributed this to users’ lack of knowledge of the YouTube community’s values. Gillespie sees in this reaction an “astounding assumption: a different logic would be to treat this new wave of flags as some kind of expression of values held by this new user population.” (P. 130.) Flagging will remain contested not just because values vary, but also because flagging relies on a devoted but small group of users to voluntarily police bad behavior, in conflict with another small group that deliberately seeks to cause mayhem.
There’s another approach, self-labeling, which the nonprofit Archive of Our Own (with which I volunteer) tries: ask users to evaluate their own content as they post it, and provide filters so users can avoid what they don’t want. This distributes the work more equitably. But tagging is time-consuming and can deter use, so commercial platforms make self-tagging limited, either relying on defaults or on rating entire users’ profiles, as on Tumblr. But self-tagging raises problems with consistency, since Tumblr users don’t always agree on what’s “safe,” not to mention what happens when Tumblr itself decides that “gay” is unsafe by definition. I’m obviously invested in the AO3; precisely because his analysis is so incisive, I wish Gillespie had spent a little time on what noncommercial platforms decide to do differently here and why.
Gillespie also has a great discussion of automated filtering. It’s relatively easy to compare images to hashes that screen out known child pornography images. But that relative ease is a product of law, technology, and policy that is hard to replicate for other issues. The database that allows screening is a high priority for platforms because of the blanket illegality of the content, and hashing known images is a far simpler task than identifying new images or their meaning in context. Microsoft developed the software, but recognized it as good PR to donate it for public use rather than keeping it as a trade secret or licensing it for profit, which isn’t true of other filtering algorithms like YouTube’s Content ID. Machine learning is trying to take on more complex tasks, but it’s just not very good yet. (After the book came out, we learned that Amazon’s machine learning tool for assessing resumes learned to discriminate against women—a machine learning algorithm aimed at harassment or hate speech will likewise replicate the biases of the arbiters who train the computer, and who tell it that “choke a bitch” is fine.) Also, we don’t really know what constitutes a “good” detection rate. Is 95% success identifying nude images good? What about a false positive rate of 1%? Are the false positives/negatives biased, the way facial recognition software tends to perform worse with darker-skinned faces?
Nor are the choices limited to removal; algorithmic demotion or screening allows people who know that the content exists to find it, while making it harder for others to stumble across it. But this makes platforms’ promise of sharing much more complicated: to whom are you visible, and when? These decisions can be hard for affected users to discover, much less understand, and they’re particularly important for marginalized groups. One good example is Tumblr’s shadow-banning of search terms like “porn” or “gay” on its app, with political consequences; similarly, it turns out that TripAdvisor reviewers may discover that they can’t use “feminism” or “misogyny” in their reviews (highlighting that algorithmic demotion always interacts with other policy choices). Meanwhile, YouTube and Twitter curate their trending pages to avoid sexual or otherwise undesired content, so it’s not really what’s trending but only an undeclared subset, which curation nonetheless never quite manages to avoid controversy or harm, as platforms rediscover every few months. Amazon does similar things with best-sellers to make sure that shapeshifter porn doesn’t get recommended to people who haven’t already expressed an interest in it. Users can manipulate this differential visibility, too, as we learned with targeted Facebook ads from Russians and others in the 2016 US presidential campaign.
Again, it might be worthwhile to consider the potential alternatives: the noncommercial AO3, like Wikipedia, has done very little to shape users’ searches, unlike YouTube’s radicalization machine.
In concluding, Gillespie judges content moderation to be so difficult that “all things considered, it’s amazing that it works at all, and as well as it does.” (P. 197.) Still, handing it over to private, for-profit companies, with very few accountability or transparency mechanisms, isn’t a great idea. At a minimum, he argues, platforms should have to be able to explain “why a post is there and how I should assess.” (P. 199.)
Gillespie suggests that Section 230 of the Communications Decency Act may warrant modification. He argues that it should still provide immunity for pure conduits and good faith moderation. “But the moment that a platform begins to select some content over others, based not on a judgment of relevance to a search query but in the spirit of enhancing the value of the experience and keeping users on the site,” (P. 43) it should be more accountable for the content of others. (I doubt this distinction would work at all.) Or, the safe harbor could be conditioned on having obligations such as meeting minimum standards for moderation, perhaps some degree of transparency or specific structures for due process/appeal of decisions. It is perhaps unsurprising that Facebook’s splashiest endeavor in this area, its “Supreme Court,” won’t provide these kinds of protections. It’s not in Facebook’s interest to limit its own flexibility in that way even if it is in Facebook’s interest to publicly perform adherence to certain general principles. This performance may well reflect sincere substantive commitments, but that also has advantages in fending off further regulation and in making bigness look good, because only big platforms like Facebook can sustain a “Supreme Court.” Although I have serious concerns about implementation of any procedural obligations (related to my belief that antitrust law would be a better source of regulation than blanket rules whose expense will ensure that no competitors to Facebook can arise), for purposes of this brief review it is probably more useful to note that the procedural turn is Gillespie’s own version of neutrality on the content of content policies. Authoritarian constitutionalism—where the sovereign adheres to principles that it announces but does not concede to its subjects the right to choose those principles—seems to be an easier ask than democratic constitutionalism for platforms.
Gillespie also suggests structural changes that wouldn’t directly change code or terms of service. He argues for greater diversity in the ranks of engineers, managers, and entrepreneurs, who currently tend to be from groups that are already winners and don’t see the downsides of their libertarian designs. This would be a kind of virtual representation that wouldn’t involve actual voting, whether for representatives or for policies, but would likely improve platform policymakers’ ability to notice certain kinds of harms and needs.
Innovation policy has focused on presenting and organizing information and content creation by users, not on innovation in governance and design or implementation of shared values; it could do otherwise. Gillespie’s most provocative question might be: What if we built platforms that tried to elicit civic preferences instead of consumer preferences? If only we knew how.
There once was a dream that was the Internet. But now the harsh morning light of Internet reality propels us to consider whether to get out of bed. Questions of content (un)trustworthiness seem omnipresent, and the liability protections of Section 230 of the Communications Decency Act are showing their age, in the opinion of their Congressional sponsors. Debates over “fake news,“ “deep fakes,” “shallow fakes,” and hybrid warfare reveal diverging ethical defaults, even among similarly situated Internet companies. Anti-vaxxers’ and medical professionals’ opinions are mistakenly considered equivalent by a portion of Internet users, and algorithms make personalized content recommendations that sometimes perpetuate false and radical beliefs. Recent indictments remind us that jurists have never resolved the question of who counts as a “publisher” on the Internet (and what duties of care that role entails). Meanwhile, machine-learning practitioners, information security experts, and other technology professionals debate the construction of shared ethical codes and professional practices. Each of these conversations inevitably implicates questions of content intermediation in technology contexts, as well as the role of “expert” knowledge, professional licensing/credentialing, and professional liability.
Claudia Haupt in Licensing Knowledge asks us to consider “whether expert knowledge is still relevant in the information age.” Answering in the affirmative, Haupt’s article offers an injection of helpful intellectual rigor into discussions of knowledge construction, expertise, and the First Amendment. Haupt engages head-on the question of the Yelpification of expertise and knowledge (and its corresponding quality control challenges) as she takes us on a thought-provoking, interdisciplinary romp into the complex issues of “expert” speech and its intersection with personalization, professional licensing, and liability. As the article explains, “[s]cholars of the legal profession have asserted that ‘[t]he Internet has provided consumers with increasing access to information about the law and to information about the quality of services provided.’” (P. 522.) Yet, the ability to judge the quality of this information presents challenges particularly because of the rise of the lay “Internet expert.” These information asymmetries impact information accuracy and warrant consideration.
After introducing the tension between the First Amendment interests of speakers and the interest of states in preserving high-quality services in commercial contexts, the article argues that expert and professional speech is part of an effort to re-calibrate existing licensing regimes. Walking the reader through caselaw spanning a wide array of “experts”—from tour guides to doctors—Haupt highlights that not all professional licensing schemes are the same, and not all occupations pose equivalent threats to health and safety. In particular, Haupt distinguishes between information and knowledge: a specific type of information communicated as professional advice. Professional knowledge, argues Haupt, is rooted in expertise formed in the context of a knowledge community and conveyed for the benefit of a particular client. Haupt explains that by adopting a framework that distinguishes information and knowledge, it is possible to reconcile the interests of speakers and listeners, thereby reconciling licensing regimes with the First Amendment.
Building on insights from the science and technology studies literature regarding the epistemological foundations of expert knowledge, the article next turns to reconciling democratization of knowledge with the challenge of maintaining information accuracy. Haupt explains that some scholars suggest that the public can rise to the level of lay experts with “experiential knowledge of a condition.” (P. 533.) However, other scholars discard the idea of a lay expert as an oxymoron, suggesting that the primary question turns on the formalized extension of knowledge and a determination of whose knowledge counts as expertise. Haupt then connects this dominant view to the idea of expertise being formed in knowledge communities. She explains that the modern idea of scientific expertise arises from two historically distinct elements: “occupational expertise and the expertise claimed by scientists as privileged knowers of truth about the world.” (P. 534.)
Haupt highlights that one distinction between professions and other occupations is the fusion of theory and practice. A claim of authority arises from the existence of a shared methodology within the knowledge community of professionals. Haupt explains that “the link between expertise and authority extends to the professions in that professional experts monopolize the ability to speak the truth.” (P. 538.) A knowledge asymmetry therefore persists, but this undemocratic reality, perhaps counterintuitively, presents the potential to advance public discourse. While the First Amendment frames expert knowledge as opinion equal to other opinions, licensing regimes by design create speaker inequality by acknowledging the role of listener interests.
Haupt explains that in the situation where both a professional licensing regime and a remedy in tort for malpractice exist, professional speech protection and licensing are actually complementary. The category of “professional speech,” argues Haupt, is thus fundamentally different and presents a “unique category of speech.” (P. 552.) It “reflects the shared knowledge of professionals” in the “knowledge community that is communicated from professional to client within the confines of a professional— client relationship.” She tells us that “bad professional advice is properly suppressed.” (P. 555.)
In a legal context, Haupt’s arguments might bring to mind the licensing/liability of broker dealers and investment advisors—a structure where compelled disclosure and regulatory oversight of personalized advice work in tandem with the dynamics of licensing, communities of practice, duties of care, and malpractice liability, without significant First Amendment concerns. Indeed, in the halcyon days of the late 1990s, the issues of personalization of investment advice, democratized access to markets, and market trust animated the SEC’s analysis of the permissibility of Internet brokerages such as E*TRADE. Thus, Haupt’s discussion offers insights that implicate a host of Internet-related trustworthiness and intermediation issues, both past and future.
The framing of Haupt’s argument around knowledge communities might also remind the reader of Michael Polanyi’s related insights on “tacit knowledge” and “public mental heritage” held by professionals in “normative dynamic orders.” Polanyi explains that “[i]n each field” generations pass on “a public mental heritage” comprised of both substantive and “tacit knowledge,” the unspoken but shared culture of “knowing how” one obtains only from being inside the community. In other words, relevant dynamic orders of experts, through consultation, competition, or a combination of the two, introduce new participants into their community of expertise. “Then, when they suggest their own additions or reforms, they return to the public and claim publicly that these be accepted by society–to become in their turn a part of the common heritage.” To wit, we might also view Haupt’s Licensing Knowledge itself as a noteworthy contribution to the public mental heritage of the legal profession.
Daniel Susser, Beate Roessler & Helen Nissenbaum, Online Manipulation: Hidden Influences in a Digital World
, available at SSRN
Congress has been scrambling to address the public’s widespread and growing unease about problems of privacy and power on information platforms, racing to act before the California Consumer Privacy Act becomes operative in 2020. Although the moment seems to demand practical and concrete solutions, legislators and advocates should pay close attention to a very timely and useful work by a set of philosophers. In a working paper entitled Online Manipulation: Hidden Influences in a Digital World, three philosophers–Daniel Susser, Beate Roessler, and Helen Nissenbaum–offer a rich and nuanced meditation on the nature of “manipulation” online. This article might provide the conceptual clarity required for the broad and sweeping kind of new law we need to fix much of what ails us. Although the article is theoretical, it could lead to some practical payoffs.
The article’s most important contribution is the deep dive it provides into the meaning of the manipulation, a harm separate and distinct from other harms more often featured in today’s technology policy discourse. Powerful players routinely deprive us of an opportunity for self-authorship over our own actions. Advertisers manipulate us into buying what we don’t need; platforms manipulate us into being “engaged” when we would rather be “enlightened” or “provoked” or “offline”; and political operatives manipulate us into voting against our interests. Taken together, these incursions into individual autonomy feed societal control, power imbalances, and political turmoil. The article builds on the work of many others, including Tal Zarsky, Ryan Calo (in an article that has received well-deserved praise from Zarsky in these pages), and Frank Pasquale, who have all written about the special problems of manipulation online.
The heart of the paper is an extended exploration into what it means to manipulate and how it differs from other forms of influence both neutral (persuasion) and malign (coercion). The philosophers focus on the hidden nature of manipulation. If I bribe or threaten or present new evidence to influence your decision-making, you cannot characterize what I am doing as manipulation, according to their definition, because my moves are visible in plain sight. I might be able to force you to take the decision I desire, which might amount to problematic coercion, but I have not manipulated you.
This insistence on hidden action might not square with our linguistic intuitions. We indeed might feel manipulated by someone acting in plain sight, and the authors are not trying to argue against these intuitions. Instead, they claim that by limiting our definition of manipulation to hidden action, we can clear up conceptual murkiness on the periphery of how we define and discuss different forms of discreditable influence. This is very useful ground clearing, helping manipulation stand on its own as a category of influence we might try to attack through regulation or technological redesign.
The piece convincingly links increased fears of manipulation, thus defined, to the current and likely future state of technology and the power of information platforms in particular. The pervasive surveillance of today’s information technology gives would-be manipulators access to a rich trove–Dan Solove’s digital dossiers and Danielle Citron’s reservoirs of danger—about each of us, which they can buy and use to personalize their manipulations. Knowing the secret manipulation formula for each individual, they can then use the “dynamic, interactive, intrusive, and personalized choice architectures” of platforms to give rise to what Karen Yeung calls “hypernudging.” Online tools hide such behavior, in the way they are designed to recede into the background; in one of the more evocative analogies in the paper, the authors argue that information technology operates more like eyeglasses than magnifying glasses, because we forget about them when we are using them. “A determined manipulator could not dream up a better infrastructure through which to carry out his plans” than today’s technological ecosystem, they conclude.
Having crafted their own definition of manipulation, and after connecting it to modern technology, the authors turn last to theories of harm. They focus on harm to autonomy, on the way manipulation undermines the ability of the manipulated “to act for reasons of their own.” We are treated like puppets by puppet masters pulling our strings; “we feel played.”
The cumulative effects of individual manipulations harm society writ large, posing “threats to collective self-government.” Consider the bolder claims of psychographic targeting made by the people at Cambridge Analytica before the last election, which if true suggest that “democracy itself is called into question” by online manipulation.
If Congress wants to enact a law prohibiting manipulative practices, this article offers some useful definitions: a manipulative practice is “a strategy that a reasonable person should expect to result in manipulation,” and manipulation is defined as “the covert subversion of an individual’s decision making.” Congress would be wise to enact this kind of law, perhaps adding it as a third prohibited act alongside deception and unfairness in section five of the FTC Act.
In addition, Congress could breathe new life into notice-and-choice regimes. Currently, we are asked to take for granted that users “consent” to the extensive collection, use, and sharing of information about them because they clicked “I agree” to a term-of-service pop-up window they once saw back in the mists of time. Were we to scrutinize the design of these pop-ups, assessing whether online services have used manipulative practices to coax users to “agree,” we might recognize the fiction of consent for what it really is. We should implicitly read or explicitly build into every privacy law’s consent defense a “no dark patterns” proviso, to use the phrase for manipulative consent interfaces by scholars like Woody Hartzog.
Finally, although these authors ground their work in the concept of autonomy, an unmeasurable concept not well-loved by economists, their argument could resonate in the god-forsaken, economics-drenched tech policy landscape we are cursed to inhabit. Manipulation, as they have defined it, exacerbates information asymmetry, interfering with an individual’s capacity to act according to preferences, resulting in market failure. A behavioral advertiser with a digital dossier “interferes with an agent’s decision-making process as they deliberate over what to buy. Worse yet, they may be enticed to buy even when such deliberation would weigh against buying anything at all.”
In fact, the authors go to lengths to explore how harmful manipulation interacts with the concept of nudges. Some nudges should count as manipulation, when their designs and mechanisms are hidden, even if they bring about positive behavioral change. The architects of the theory of nudges might even embrace this conclusion. The article quotes liberally from Cass Sunstein, who has explored the ethics of government-imposed nudges, acknowledging their sometimes manipulative quality. The article resonates with recent ruminations by Richard Thaler, who has coined a new term, “sludges,” the negative mirror image of positive nudges. These fathers of nudges are finally cottoning on to what privacy scholars have been writing about for years: at least online, the negative sludges we encounter seem to outnumber the positive nudges, with the gap widening every day.
We have a new target in our sights, whether we call them manipulative practices, dark patterns, or sludges: the technological tools and tricks that powerful information players use to treat us like their puppets and cause us to act against our own self-interest. By lending precision to the meaning of manipulation, this article can help us meet the challenge of many of the seemingly impossible problems before us.