Jan 4, 2023 Daithí Mac Síthigh
“Retrofuturism” in art and literature is a look back at the (sometimes recent) past and how the stories of the future were told. The retrofuturist aesthetic can be found in present-day theme parks like Walt Disney World’s Tomorrowland and EPCOT and in the concept of steampunk. Through retrofuturism, we try to understand what was once hoped for, often as a way of understanding success or failure and of critiquing present-day efforts and priorities.
Retrofuturist impulses are particularly important in technology law scholarship. Critical appraisals of ‘smart city’ and urban innovation projects and initiatives examine how people joined the digital with the material to imagine a better world. You can’t tell the story of the smart city without at least engaging with the tales of the city. And so, in a very real and immediate way, the literature of geography, planning, and–yes–physical architecture is a key resource for the legal scholar. In The Kind of Solution a Smart City Is: Knowledge Commons and Postindustrial Pittsburgh, Michael Madison gives us a compelling retrofuturist account of Pittsburgh, the smart city. Madison’s account of a range of projects in Pittsburgh (including those of the 21st century) tells a story that is both universal and particular, tapping into the need to understand the roads taken and not taken, and what was imagined or foreseen in the recent and not so recent past.
The Governing Knowledge Commons framework, an approach to which Madison himself has made founding and abiding contributions, allows for the study of how intellectual and cultural resources (e.g. information, science, and software), as distinct from natural resources, are created and shared, and in turn governed through, by, and with communities. Applying this framework, Madison digs into the past ideas and initiatives meant to improve (or “fix”) this mid-sized Pennsylvanian city. Imagined overlapping and data-driven futures like the pioneering Pittsburgh Survey of a century ago, the 3RC (Three Rivers Connect) civic computing initiative of 1999, or reaching the final of (but not winning) the USDoT Smart City Challenge in 2014 offer rich resources for anyone seeking to understand how cities attempt to anticipate and evolve in the face of disparate and dynamic challenges.
Madison also tells the stories of the conditions for urban reform and renewal in Pittsburgh, contributing to the overall argument that context, geography and history all matter. Physical infrastructure is old. Social and political infrastructures are tied up in long-standing institutions and networks. Pittsburgh’s population has been declining. The geography of the city, in its ups, downs, rivers, and bridges, is irregular. The city’s many neighbourhoods are disconnected from political power. The City Council is not the only game in town, due to the presence of regional and other government structures. Where once there was steel, now there are universities and hospitals (“eds and meds”) and, increasingly, an innovation economy (“tech-centred development”). Yet until recently, data systems weren’t often used in municipal government.
Readers will recognize many of these facets in “post-industrial” cities around the world. Pittsburgh is one of a number of cities where ‘economic renewal efforts’ dominate cultural and political discourse, decades after the decline of a major production or extractive industry. But, as Madison makes clear, Pittsburgh is unique in how its infrastructure, population decline, geography, and changing industry are interconnected with money, power, and people, meaning that the factors and actors affecting economic renewal and the digital transition require particularly close attention.
Some political institutions and funders nowadays emphasise the “knowledge square” (e.g., as the European Commission now puts it, the interconnection between education, research, innovation, and service to society). Though Madison does not put it quite this way, his careful attention to the roles of philanthropic organisations (a distinctive part of the Pittsburgh civic story) and universities tells another dimension of Pittsburgh’s reform and development trajectory. He highlights the different roles played by the University of Pittsburgh and Carnegie Mellon University and its projects. These include the smart cities institute Metro21. Madison also traces the distinctive character of a number of major interventions, such as the Western Pennsylvania Regional Data Centre, and the individuals who have led and championed them.
This article is not (only) a celebration of a great city, though. Madison highlights the difference between the problems the city tries to solve and the biggest problems that need to be solved. He retains an appropriate scepticism about extreme smart city boosterism, calling for greater attention to evolution over creation and to the enduring role of physical infrastructure and its limits.
The dream of a better tomorrow is at the core of urban (re)imagination. As Pittsburgh moves towards being a smart city, Madison draws a contrast between the city’s “older smoky self” and its aspirations towards becoming an “equitable and forward-looking ‘green’ community”. Yet Madison has also shown us that the smoke never fully clears. Even the ‘recent’ history of what was tried and why it did or didn’t work in the late 1990s is in his account an essential part of a proper understanding of the choices now available to this particular city. Other cities will face different physical and political factors, but Madison is rightly calling on us to map and understand those local conditions; some common questions, but different answers, are what we can hope to find.
Nov 28, 2022 Stacy-Ann Elvy
Consumers accessing goods and services online are inundated with numerous disclosures, privacy policies, end user license agreements and terms and conditions. In connection with the so-called “duty to read,” consumers have historically been presumed and expected to fully review contract terms as part of the contract-making process. Yet, as several scholars have observed, consumers do not appear to consistently review contract terms: what some have called the “no-reading problem.” The failure of consumers to review and understand contract provisions before manifesting assent may incentivize companies to offer one-sided contracts with terms that are primarily beneficial to businesses.
In their new article, Contracts in the Age of Smart Readers, Professors Yonathan A. Arbel and Samuel Becher make a noteworthy contribution to scholarship in the technology and contract law fields by highlighting how nascent technological advancements in language models associated with artificial intelligence can disrupt the status quo. Their powerful article adds to an existing body of scholarship exploring the important connection between technological developments and what the authors describe as one of the underlying justifications for legal intervention in consumer transactions: the “no reading problem.”
Arbel and Becher tout various possible benefits of novel language models, which they label as “smart readers,” by offering several examples of this technology in action. They observe that armed with a smart reader app, a consumer could in theory use their smartphone to scan and receive a plain and concise explanation of boilerplate provisions in a company’s terms and conditions. Contractual text could be personalized based on the needs of each reader by factoring in cognitive, linguistic, and cultural patterns. A consumer using a smart reader could request concrete examples describing the possible implications of boilerplate clauses.
Arbel and Becher note that smart readers have the capacity to compare the terms of a company’s privacy policy with those offered by other businesses and generate an industry score that the consumer could then use to comparison shop. The authors convincingly argue that, if widely adopted, this technology could potentially enhance consumer understanding of contract terms and privacy policies and the risks associated with the same, as well as increase consumer awareness of market alternatives. They contend that smart readers may facilitate “term competition” (P. 91) in certain markets, even if the technology is not widely adopted.
After persuasively describing the potential advantages of smart readers, Arbel and Becher highlight the possible risks associated with smart readers. These concerns include the possibility of courts over-relying on consumer access to such apps, which may negatively impact outcomes for consumers. Adversarial attacks, which the authors describe as “a method of exploiting the statistical nature of machine learning models” (P. 121) may also make contractual explanations and industry scores less accurate and reliable. Arbel and Becher note that in some cases smart readers could oversimplify boilerplate terms, which could decrease consumer understanding. Lastly, businesses could offer better terms to those consumers who they believe will use smart readers and comparison shop, and less favorable terms to those who do not, thereby exacerbating discrimination concerns.
Arbel and Becher posit that legal interventions in favor of consumers are often “couched in the no-reading problem.” (P. 134.) However, smart readers offer a different way of tackling the no-reading issue. They suggest that the no-reading problem is perhaps a technological issue that smart readers can help to solve, rather than an ethical one deserving of legal intervention. The authors contend that while smart readers do not address various other justifications for pro-consumer legal intervention, such as other forms of market failure, smart readers may soon render the no-reading justification obsolete. Arbel and Becher’s notable and insightful description of smart readers’ growing potential should be of particular interest to technology law, contract law, and consumer law scholars, as well as others who are interested in learning more about the ways in which technological advancements may impact core justifications for consumer protection intervention.
Nov 2, 2022 Paul Ohm
In the aftermath of the Cambridge Analytica fiasco, Facebook was pummeled by legislators, regulators, and advocates around the globe for their poor privacy practices stemming from the way the company seemed to prioritize growth and profit over other all else. As one small part of a multipronged defense, the company hired four prominent privacy advocates, former fierce critics of the company. The early evidence suggests that these four—and other likeminded Facebook employees—haven’t had much success reorienting the company. As one data point, two years after they were hired, Frances Haugen blew the whistle on how Facebook had not done enough to weed out misinformation, combat threats to democracy, and protect vulnerable teens, again due to a relentless pursuit of growth. To be fair, the Haugen story isn’t only or primarily a privacy fiasco, but it belies the idea that good people in positions of authority have helped the fix the company from within.
This isn’t just a Facebook story. Every large technology company employs people who profess to be privacy advocates in positions of authority, yet their collective efforts do not seem to have had done much to alter the troubling trajectory of their employers’ products and services. Ari Waldman, the deeply interdisciplinary privacy law scholar from Northeastern University, has written a vital and important book investigating why bad privacy outcomes occur at firms that employ well-meaning and well-trained privacy professionals. Drawn from dozens of interviews with software engineers and privacy professionals from many technology companies, Waldman presents a compelling and distressing picture, revealing the way companies constrain the influence of privacy-focused employees, repurposing their work toward serving data extractive goals, eventually redefining privacy law itself in narrow, compliance-focused terms.
A trained sociologist and legal scholar, Waldman conducted 125 interviews over four years and insinuated himself into product design meetings, industry conferences, and company breakrooms, revealing a rigorous and detailed description of the way privacy is subverted and denied inside these companies. The work builds on and pays due credit to the groundbreaking qualitative work of Deirdre Mulligan and Ken Bamberger, the famous “privacy on the ground” study from a decade ago, even as Waldman offers a respectful corrective, pushing back on many of the sunnier conclusions of the earlier work.
Waldman’s conclusions are layered and sophisticated and hard to do justice to in a short review. Technology companies deploy a “coercive bureaucracy,” multiple strategies designed to limit privacy reforms and to disempower privacy professionals. One key mechanism of the coercive bureaucracy is “managerialism”, borrowing from Julie Cohen (who in turn borrowed from Judith Resnik and others), meaning the cynical transmutation of laws like the GDPR and CCPA from obligations designed to protect consumers into narrow compliance measures focused on limiting liability and deflecting regulator attention, in some cases essentially inverting these laws to require nothing that might impede the company’s growth and revenue goals.
Managerialism is but one tool of the coercive bureaucracy, and Waldman identifies too many others to list comprehensively, but to highlight a few: privacy gets redefined to being about giving users control over their personal information. (Chapter 2 is an amazing primer of the vast literature making this argument.) Privacy gets translated into narrow, codeable targets, such as finding new places to apply encryption. Privacy is what you outsource to growing armies of GDPR and CCPA consultants.
Although Waldman has written a book for scholars, it will also prove useful to privacy professionals who might recognize the disconnect between the hard work they are doing and the poor privacy outcomes their companies are producing. Chapters 5 and 6 read like how-to guides for stuck privacy professionals, building from the micro to the macro. At the individual level, Waldman surveys the subtle, small “traps” that companies use to constrain the influence of their workers, such as the “expertise trap,” which siloes people into narrow lanes of expertise, or the “access trap,” the belief that advocates should choose their battles rather than complain about every privacy transgression lest they be cut out of the decisionmaking loop. Waldman’s book will help those living inside a coercive bureaucracy spot, and maybe resist, the mechanisms constraining their work.
Ultimately, Waldman does not believe that individual awareness and resistance will be enough. Chapter 6 is a broad call to action, if not revolution, to recruit privacy professionals into a new movement, one that might serve as a “counterweight to corporate power,” the chapter’s oft-repeated mantra. He outlines fixes for privacy discourse, privacy law, and privacy organizing, to help us find new ways to break coercive bureaucracies. He makes several explicit calls to the labor movement, at one point calling for the formation of a new union of privacy workers.
There is so much I like (lots!) about this book. It provides deep, rich, and rigorously gathered empirical data about the forces that keep privacy at bay inside technology companies. It synthesizes these observations into compelling explorations of the mechanisms at play. It engages deeply and efficiently with multiple vast literatures, making it a readable and concise recommendation for newcomers to the field. I have recommended Chapter 2 to anybody still under the thrall of the consent-and-control model of privacy law; Chapter 3 to the staff working for state regulators drafting privacy rules; and the entire book to those trying to operationalize Julie Cohen’s theories. It offers multiple concrete prescriptions on how we might do better, ranging from the narrowly practical to the audaciously ambitious. It does all of this in crystal clear prose, studded with quotes and conversations from the empirical work, and suffused throughout with the considerable humanity of the author. It’s a welcome and rightful new inductee into the canon of privacy law, a must-read for students, scholars, policymakers, and privacy professionals.
Oct 26, 2022 Rebecca Crootof
I always love scholarship that forces me to pause and question my baseline assumptions. And so—as someone who has written of the need to close accountability gaps associated with malicious cyberoperations, IoT devices, and autonomous weapon systems—I was delighted to read John Danaher’s Tragic Choices and the Virtue of Techno-Responsibility Gaps. In this work, Danaher challenges everyone who has ever argued that new technologies problematically undermine traditional accountability structures by quietly observing that these new gaps are…maybe sometimes a good thing?
While Danaher tends to focus more on moral responsibility than legal liability, if you are a techlaw scholar thinking about accountability gaps in any context, add this to your reading list. Danaher writes in a relaxed and engaging style, includes a fantastic literature review of non-legal texts on accountability gaps, and explores a counterintuitive argument—all in a piece that clocks in at a svelte 22 pages of text. (Would that I could accomplish so much, so smoothly, in so few words!)
Danaher defines a “Techno-Responsibility Gap” as follows: “As machines grow in their autonomous power (i.e. their ability to do things independently of human control or direction), they are likely to be causally responsible for positive and negative outcomes in the world. However, due to their properties, these machines cannot, or will not, be morally or legally responsible for these outcomes. This gives rise to a potential responsibility gap: where once it may have been possible to attribute these outcomes to a responsible agent, it no longer will be.” Danaher then distinguishes the various forward- and backward-looking forms techno-responsibility gaps might take. There are (1) accountability gaps, which exist when there’s no one to provide a public account for the harm; (2) culpability gaps, which exist when there’s no one to take the blame; (3) compensation gaps, which exist when there’s no one to pay for the harm; (4) obligation gaps, which exist when there’s no one who ensures the harm is avoided; and (5) virtue gaps, which exist when no one takes responsibility for the harmful acts. Danaher also notes the distinction between positive responsibility (“Great job there!”) and negative responsibility (“Why didn’t you . . .?!”).
Danaher then summarizes familiar proposed means of eliminating these gaps, most of which boil down to justifications for ascribing accountability to a prescribed human or non-human entity. He concludes that, for all of the disagreement around how best to address them, “most contributors to the techno-responsibility gap debate tend to agree on one thing: the creation of techno-responsibility gaps is a problem.” Why? Because responsibility is always a good thing. Right? Right?
Maybe not! To set up his argument for why we might sometimes want to prioritize other goals over ensuring accountability, Danaher starts with the problem of tragic choices. Human decision-makers often confront questions where moral considerations simultaneously weigh in favor of different answers and it is difficult or even impossible to reach a morally comfortable conclusion. We all face these choices in our daily lives. (Do I give my discretionary funds to this or that charity?) But they become policy questions when we need to determine how best to allocate scarce resources (Should hospitals privilege this or that type of patient when deciding who receives a needed ventilator?) or weigh costs to X against costs to Y (How to balance a right to speech against a right not to be threatened?).
When confronted with these tragic choices, we—as individuals, as institutions, or as societies—may handle the moral difficulty of reaching a conclusion in various ways. First, we might delude ourselves into believing it’s actually an easy question (“illusionism”). This can manifest in ignoring costs, compartmentalizing them, or rationalizing them away. Second, we might delegate the choice to another (“delegation”). We do this when we ask waitstaff what we should order, look to a panel of judges to decide the scope of a law, or flip a coin to determine our next course of action. Third, we might make a decision and bear the psychological costs ourselves (“responsibilization”).
One of Danaher’s main points is that none of these responses will always be better or worse than the others. Rather, in a classically lawyerly move, Danaher maintains that the preferable response will depend on the situation and context. Despite our collective bias towards responsibilization, each of these responses has distinct benefits and drawbacks.
Illusionism permits mental comfort at the expense of honesty. Delegation allows for shifting the psychological and moral costs to a (possibly more informed, capable, or impartial) substitute actor. But it also risks a concentrated group or institution bearing these costs, transference to an inept decision-maker and consequently poor outcomes, and the failure of the original actor to develop or maintain decision-making skills. Finally, responsibilization enables moral agency and all sorts of accountability—but does so at the possible cost of unjustly transforming decision-makers into scapegoats. (This point reminded me of an argument against including steering wheels in fully autonomous vehicles: the idea was that, in the event of a deadly crash, the human operator would unfairly blame themselves for not intervening despite not being able to act with the reflexes necessary to prevent the accident.)
If each response to a tech-fostered accountability gap has distinct pros and cons, there will necessarily be situations where delegation will be preferable to responsibilization. Further, Danaher argues, the possibility of delegating to an algorithm, rather than to another human, may change the balance of benefits and harms associated with these different responses, insofar as it eliminates the delegation drawback of concentrating the psychological and moral costs of tragic choices with few individuals. To take advantage of this reduced cost on human decision-makers, Danaher concludes, we must be willing to live with some techno-responsibility gaps.
Danaher suggests that human online content moderators provide an example of when this tradeoff might be worthwhile. These decision-makers have a stressful, difficult job; they save untold numbers of platform users from having to view offensive and traumatizing content, but they do so at great psychological expense. Assuming both human and algorithms performed moderation tasks equally well, transferring content moderation decision-making power to an algorithm would minimize harm to humans. Similar arguments could be made for drone operators, police body-cam reviewers, and any other human charged with sifting through painful content to determine what can be cleared for public release.
Danaher is quick to qualify his argument. To the extent they are made, delegations should be made carefully; his analysis does not suggest that we should always delegate decisions to machine systems. And the fact that algorithmic decision-makers reduce some of the costs of delegation does not mean they eliminate other costs; there are still plenty of reasons to be wary of accountability gaps. Danaher also engages, in a wonderfully non-defensive manner, with various alternative versions and critiques of his argument. He explores a proposal to employ randomization as a low-cost form of algorithmic delegation, the concern that delegation fosters agency-laundering and liability evasion, and a query as to when we might (and might not) want to make the costs and tradeoffs inherent in tragic choices more explicit.
This thoughtful, dense, yet accessible piece invites readers to question our assumptions about why we assume accountability—and, specifically, responsibilization—is always preferable to the alternatives. I will likely to continue to argue for closing tech-fostered accountability gaps, but thanks to this piece, my arguments will now be far more nuanced.
Sep 23, 2022 Rebecca Tushnet
“Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” This statement is often attributed to retail mogul John Wanamaker. In his provocative new book, Subprime Attention Crisis: Advertising and the Time Bomb at the Heart of the Internet, Tim Hwang argues that online, Wanamaker’s statement is far too optimistic. This slim volume is packed with infuriating details about how the immense, opaque, and inescapable machinery of online advertising exists primarily to move massive amounts of money to intermediaries, many of them fraudulent or fraud-indifferent. The benefits to consumers and publishers have been minor and incidental; the harm to democracy has been severe.
Hwang argues that the current internet is built on a fundamentally flawed model of ad-supported content. He explains that there are no real checks on online advertising fraud; there are too many intermediaries for advertisers (or publishers) to police in any coherent way. Ad failure online is pervasive. A publisher may claim it displayed ads online but consumers nevertheless may not have seen it due to poor placement (such as at the bottom of the page). Hwang notes, “[i]n 2014, Google released a report suggesting that 56.1 percent of all ads displayed on the internet are never seen by a human.” (P. 81.) Even when real humans are involved, they may not be the actual audience for ads; Hwang cites research that up to 50 percent of all click-throughs on mobile devices may be the result of accidental “fat finger” clicks. (P. 79.) He also notes fraud estimates of up to $1 out of every $3 spent on digital advertising, including ads served to devices that were “not real phones at all, or … were phones running automated scripts, unseen by any actual members of the public.” (P. 85.) (This isn’t just phones; it seems to be a pervasive problem with all digital advertising, including digital TV ads.) Even with exculpatory contracts, some of these problems have spilled over into litigation. But litigation will never catch up with today’s problems, even if (implausibly) it provided a full remedy for past harms.
The existing system involves too many intermediaries to provide any certainty that real people, much less the right people, are seeing the ads supposedly targeted to them, even before the ever-increasing presence of ad-blockers is taken into account. Rather than publishers being ad-supported, mostly it’s intermediaries sucking out as much as 70 percent of the revenues from an ad. Hwang’s critique of targeted advertising—it’s way more expensive than nontargeted ads and often quite inaccurate and permeated with fraud—is persuasive for anyone looking for how they should allocate their ad budgets.
But in the broader sense, I worry that Hwang is shouting into the void. Along with Wanamaker’s quote, I kept thinking of two other phrases as I read the book:
- “If something cannot go on forever, it will stop.”
- “This time is different.”
Hwang’s model of the “subprime attention crisis” makes a clear analogy to the subprime mortgage bubble, which collapsed and did huge amounts of harm—financial and otherwise—to millions of people (though largely not to the firms deemed too big to fail, who had profited from the bubble, and the principals of those firms). Hwang foresees a potential collapse of the online advertising market, taking with it the people who are actually doing journalism and otherwise creating expression other people would like to see.
But a bubble can last a lot longer than it should when there don’t seem to be other, better alternatives, which may be the case here. Given the constant renaming and tweaking of digital business models, the infinite promise of revised algorithms, and the rise of new platforms—Tiktok and its influencers, for example, don’t appear in the book—it is always possible for advertisers to hope that this time is different. Fundamentally, advertisers have budgets and want to use them, and while the occasional giant like Procter & Gamble can give up on big online advertising venues because of its enormous brick and mortar footprint (and its other advertising options), most advertisers can’t and won’t. Individual publications, from the New York Times to a Dutch public broadcaster are experimenting with promising advertising-funded models that cut out more intermediaries. And yet cutting out intermediaries generally means not scaling up, putting inherent limits on those alternatives, especially for new market entrants. If those initiatives expand, they will then become intermediaries for other publications, with some of the same risks and pressures (though perhaps a better ethical compass or at least an incentive to divert more money into publishers’ pockets).
So should we give up on preventing ad fraud and the marketplace distortions it causes? Not at all. But we also shouldn’t expect that a market correction will take care of the problem. Online ad money moves in ways so fast and complicated that Dina Srinivasan has suggested regulating ad exchanges the way we regulate financial exchanges for transparency and fairness. But if, as Hwang persuasively argues, lots of the money involved is just stolen by people who don’t provide the promised services, we should be thinking not just about transparency/antidiscrimination rules but anti-money laundering, know-your-customer, and anti-fraud regulations. Mark Lemley has pointed out that a lot of “fair access” rules are also “access for fraudsters/bad guys” rules, making platform regulation extremely hard to get right. Given that fraudsters and bad guys aren’t having much trouble accessing the current online advertising regime, however, Hwang’s book strengthens the case for looking much more aggressively under the hood of online advertising systems, using both advertisers’ purchasing power to require better-audited results and regulators’ power to set enforceable rules.
I would be remiss not to note the risks of regulatory intervention. Texas’s Attorney General is right now pursuing Twitter for, ostensibly, understating the number of “bot” accounts in order to make Twitter more commercially appealing. But it is obvious that he is doing so to punish a perceived political enemy and to reward a newly announced Republican billionaire supporter, Elon Musk. Nonetheless, fraud is fraud, and the risks of biased enforcement have to be balanced against the current freedom of private parties to steal from people who produce valuable things, including the news, with apparent impunity. We can’t start making the necessary decisions without understanding the scope of the problem. Hwang’s book helps with that.
Aug 11, 2022 Natalie Ram
American law typically treats privacy and its associated rights as atomistic, individual, and personal—even though in many instances, that privacy is actually relational and interdependent in nature. In their seminal article on The Right to Privacy, for instance, Samuel Warren and Louis Brandeis described privacy as a “right to be let alone.” Doctrines of informed consent are generally concerned with “respect[ing] individual autonomy,” even as the information disclosed or withheld by that consent may implicate the privacy of others. Similarly, consumer genetics platforms seek authorization from a single individual before processing or uploading a genetic profile, even though law enforcement now routinely searches those profiles to identify distant relatives who may have committed prior criminal acts.
In their article, Privacy Dependencies, Solon Barocas and Karen Levy move beyond the observation that privacy is relational to provide a typology of the “varied ways in which one person’s privacy is implicated by information others reveal.” They identify three broad types of privacy dependencies: those based on our social or other ties (tie-based dependencies), those drawn from our similarities to others (similarity-based dependencies), and those revealed by our differences from others (difference-based dependencies). While social norms or legal obligations may serve to discipline some of these privacy dependencies, they will be inapplicable or inapposite for many others. Barocas and Levy masterfully survey the wide range of normative values and diverse areas of law that may be affected by privacy dependencies. Taking genetic data as a case study, Barocas and Levy then demonstrate how each form of privacy dependency can arise in this context—and how each has been exploited in criminal investigations. They conclude that a greater attentiveness to privacy dependencies, and when and how they arise, can inform better policymaking and give us greater purchase on the values that privacy serves.
Barocas and Levy devote the bulk of their article to identifying and explaining each of the three forms of privacy dependencies that make up their typology, subdividing each into several subtypes. The first category of privacy dependencies, tie-based dependencies, exploits information gathered about one individual (Alice) to learn about another individual (Bob) by virtue of some relationship between them, whether known or unknown to Alice and Bob themselves. Barocas and Levy further subdivide this category into four types. A “passthrough” is a tie-based dependency in which Alice passes information about Bob on to some observer, or Alice and Bob share information through some third-party intermediary like Facebook or Gmail. A “bycatch” occurs where information about Bob is incidentally, but foreseeably, collected in the process of learning about Alice, as with police body-worn cameras. “Identification” can turn on a tie-based dependency, as where an unknown Bob can be identified due to his connection to a known Alice. Finally, “tie-justified dependencies” exploit social ties between Alice and Bob to justify expanding surveillance from Alice alone to also include Bob.
The government has exploited each of these forms of privacy dependency in national security and criminal investigations as, for instance, in the investigative use of consumer genetic data to target genetic relatives as suspects or the National Security Agency (NSA) bulk telephony metadata program. So too have social media entities as, for instance, in the Cambridge Analytica scandal at Facebook or Amazon Ring’s surveillance devices. Troublingly, for the most part, the law has not vested individuals whose privacy is affected by a tie-based dependency with protections against these kinds of privacy losses. Indeed, key Fourth Amendment doctrines encourage the government to exploit our interdependent data privacy. Moreover, social norms may be of limited utility in regulating against unwelcome exposure, particularly where the tie being exploited is involuntary or unknown to its subjects.
The second category of privacy dependencies that Barocas and Levy identify is based on similarity, in which information that Alice discloses about herself may be imputed to Bob insofar as Bob “is understood to be similar to Alice.” This form of dependency may turn on three ways in which individuals may be “similar” to others: based on “the company you keep”; on some “social salient characteristics that you share with others (e.g., gender, race, and age), but with whom you hold no explicit social ties”; or more distantly, on “non-socially-salient” characteristics, as in behavioral advertising.
Insurance is a paradigm example of similarity-based inference at work, but these dependencies may also arise in the context of criminal law (where bail, sentencing, and other decisions may turn in part on statistical risk assessments tools), credit scoring, advertising, and others. As Barocas and Levy observe, “[s]imilarity-based dependencies violate the moral intuition that people deserve to be treated as individuals and subject to individualized judgment.” And yet, “there is no way to avoid using generalizations or avoid being subject to them.” Moreover, similarity-based dependencies may be troubling both “when they subject people to coarse generalizations” and “when they allow for overly granular distinctions.” Particularly when they depend on non-socially-salient characteristics, similarity-based dependencies may fail to elicit the social solidarity that might restrain the excesses of this data inference mechanism.
Finally, difference-based dependencies arise when, by revealing some information about herself, Alice enables an observer to learn something about Bob by making herself distinguishable from him. Here, too, this dependency may occur in three ways: by “process of elimination,” in which Alice’s disclosure makes an unknown Bob’s ultimate identification more likely; by “anomaly detection,” in which Bob’s atypicality becomes apparent by comparing his data to that of many “normal” Alices; or by “adverse inference,” in which Bob’s refusal to disclose some information appears more suspect because most Alices disclose. Importantly, unlike tie-based and similarity-based dependencies, none of these forms of difference-based dependency requires a prior connection between Alice and Bob. Moreover, there is little Bob can do to protect his privacy in these cases. As Barocas and Levy observe “any attempts he might make to do so may, perversely, make him stand out even more.” The difficulty of this kind of dependency is evident in the NSA’s approach to encrypted communications, which has treated the fact of encryption itself as a basis for retention and analysis.
For these difference-based dependencies, collectivity is “essential to privacy preservation here.” Yet collective action may be difficult to muster where individuals may be “unaware of the effects of their disclosures or acting out of requirement or self-interest.” Instead, difference-based dependencies, Barocas and Levy conclude, are best restrained by restricting mass data collection in the first instance, since difference becomes apparent only in comparison to many others.
The payoffs for Barocas and Levy’s detailed typology of privacy dependencies are several. For one thing, as Barocas and Levy explain in a case study of privacy dependencies in genetic data, statutory protections may yield unexpected privacy dividends, where a protection adopted with one type of dependency in mind may come to protect against manipulation of another. Consider the Genetic Information Non-discrimination Act (GINA), which, although enacted as an anti-discrimination statute, has demonstrated value as an employee-privacy statute as well. Barocas and Levy also describe myriad ways in which law enforcement has exploited privacy dependencies in the context of genetic data. In so doing, as Barocas and Levy observe, identifying the various privacy dependencies at work can “help us determine if and when we even recognize Bob as a party with a legitimate privacy claim,” “shed light on the varied normative goals that we expect privacy to serve,” and “suggest possible targets for intervention.”
Perhaps most forcefully, Barocas and Levy provide a further perspective on the inadequacy of notice-and-choice as a paradigm for privacy regulation. As they explain, “[i]f we are scarcely able to make decision that attend to our own privacy interests, the goal of recognizing shared interests should not be to further burden our individual choices with an expectation that we take into account the interests of others.” And they conclude that “[r]ecognizing the mechanisms that create different forms of dependency does more than demonstrate the shortcomings of privacy individualism; it lays the groundwork for well-tailored policymaking and advocacy.” Ultimately, Barocas and Levy give an irrefutable accounting of the many ways in which individualism fails privacy, and their typology for organizing and understanding these failures make better privacy law possible.
Jul 13, 2022 James Grimmelmann
Gregory Klass,
How to Interpret a Vending Machine: Smart Contracts and Contract Law, 7
Geo. L. Tech. Rev. __ (forthcoming, 2022), available at
SSRN.
Gregory Klass’s How to Interpret a Vending Machine: Smart Contracts and Contract Law is an extraordinarily incisive legal analysis of smart contracts. While others have written insightfully about the relationship of smart contracts and legal contracts, Klass utterly nails a central conceptual point: When smart contracts are embedded in legal relationships, they stand in need of interpretation.
Nick Szabo introduced smart contracts in the 1990s as contracts “embedded in the world” such that breach is expensive or impossible. Whereas traditional contracts rely on the legal system (backed by threat of force) to enforce their terms, smart contracts use hardware and software to automatically enforce their terms. Szabo gives the example of a “humble vending machine” that takes in coins and dispenses products, and then argues that software and cryptography make it possible to craft much more sophisticated agreements than simple cash sales.
Seen this way, smart contracts are not contracts but mechanisms, and hence the vending-machine analogy is apt. What is important is not what they mean but what they do. Klass shows that even mechanisms need interpretation. Through a sequence of entertaining hypos, he demonstrates that courts confronting cases involving mechanisms embedded in contracts must use the methods of legal interpretation to reason about what those mechanisms are understood to do, just as they reason about what contractual text is understood to do.
His examples, cleverly, also involve the humble vending machine. He starts with a “standard,” “black-box” vending machine: Charles Kingsfield, Jr. inserts two quarters into an Acme Vending Company machine, and a hidden mechanism dispenses a sugary snack. If the machine jams and the snack fails to fall, this is a breach of a contract of sale and Kingsfield is entitled to a refund. How do we know it’s a contract of sale governed by the UCC? From the “shared cultural understanding of what vending machines do.” That shared cultural understanding doesn’t depend on the specifics of the gears and cams in the guts of Acme’s machine; instead, it has to do with how the form of the vending machine itself communicates an offer to engage in cash transactions for the sale of snacks.
Next, Klass invites the reader to consider a “glass-box” vending machine: mutually designed and constructed by two contracting parties to implement their exchange. Orville inserts chocolate truffles into the machine, and Wilbur inserts cash, and the machine duly delivers truffles to Wilbur and cash to Orville in corresponding quantities. Klass persuasively argues that now, the internal workings of the vending machine (and not just its appearance as a vending machine) are relevant. As he puts it, “the design of Orville and Wilbur’s mutually constructed vending machine belongs to the interpretive evidence of their agreement.” If the workings of the machine reveal that it was constructed to dispense thirteen truffles every time Wilbur inserts enough money for twelve, a court could reasonably conclude that Orville and Wilbur’s contract was for baker’s dozens.
This point leads Klass to a well-taken distinction between the design and the operation of the machine. If the bill scanner occasionally reads $20 bills as $10 bills and dispenses only half as many truffles as a result, Orville must deliver the balance of truffles or give Wilbur a refund for the extra $10. But if the machine itself is deliberately constructed to occasionally deliver only half as many truffles as paid for, then this behavior was contemplated by the parties and forms part of their agreement. As a programmer would say, the contractual meaning of a glass-box mutually constructed mechanism is determined by its features, not by its bugs.
Klass’s third example, and the most vexed, involves glass-box vending machines that are unilaterally constructed. We are back to Acme again, except that now there is a large glass panel on the side of the machine that allows users to inspect the details before they insert their coins. Most of the time, this makes no difference. Professor Kingsfield can’t even remember the names of his students; he can hardly be expected to inspect the inner workings of every glass-sided vending machine he encounters. If the machine eats his quarters, Acme owes him a refund regardless of whether the quarter-eating is a deliberate feature or an unintentional bug.
This said, Klass argues that there might be a few circumstances under which a purchaser should be held to have agreed to non-standard vending-machine behavior. Perhaps, for example, the machine is installed in the lounge of the mechanical engineering department and comes with a prominent disclaimer warning users to inspect the mechanism closely to understand what it does. But these cases will be rare. Under most circumstances, the mechanism is not relevant evidence as to the understanding of parties who have not participated in its design.
The genius of the article is that the argument goes through almost perfectly unchanged when the implementation moves from vending machines to blockchains. Interwoven with its discussion of hardware-based vending machines is an equally sophisticated analysis of software-based smart contracts. There are circumstances under which a smart contract’s code —its text and not just its effects — is necessary to understand the terms of a contract involving it. But these circumstances are neither “never” nor “always,” so even the question of whether to interpret the code is context-dependent.
Klass’s thought experiments show that what matters is the relationship a mechanism and its users: not the technical details themselves but how users understand or should be expected to understand them. This viewpoint, which is characteristic of legal institutions in general and of contract law in particular, connects to Karen Levy’s important work on “book-smart” contracts, in which she shows that smart contracts fail to capture the ways in which “people use contracts as social resources to manage their relations.” Klass’s reflections about “what legal contracts do” built on Levy’s insights to argue that the “techno-utopian dream of governance by code rests on an anemic view of human sociability.”
This perceptive article is of obvious interest to scholars working on blockchain- and Internet-related issues. But its creative application of contract doctrine should also be of interest to contract theorists and legal philosophers. Klass offers important insights about the nature of bargained-for exchange in a world of mechanisms, and about the nature of legal interpretation when the texts to be interpreted are written in a programming language rather than in a natural language. This is the smartest analysis of smart contracts I have read.
Cite as: James Grimmelmann,
The Humble Vending Machine, JOTWELL
(July 13, 2022) (reviewing Gregory Klass,
How to Interpret a Vending Machine: Smart Contracts and Contract Law, 7
Geo. L. Tech. Rev. __ (forthcoming, 2022), available at SSRN),
https://cyber.jotwell.com/the-humble-vending-machine/.
Jun 13, 2022 Nicholson Price
Rachel Sachs,
The Accidental Innovation Policymakers, __
Duke L.J. _ (March 27, 2022 draft, forthcoming 2022), available at
SSRN.Innovation policy is hard. Getting it right requires balancing incentives for developers, consumer access, rewards for later innovators, safety concerns, and other factors. This balance is vitally important and wickedly difficult—even when it’s the focus of concerted, careful, informed effort. How well should we expect it to go when innovation policy is made by accident?
Enter The Accidental Innovation Policymakers, an illuminating new project by Professor Rachel Sachs. Sachs persuasively shows how Congress has repeatedly made substantial changes to innovation policy, seemingly without talking about, seriously considering, or even recognizing that it is doing so. There’s an asymmetry to this accident, and it favors industry. When Congress wants to directly promote innovation, it explicitly gives rewards to the biomedical industry. When Congress focuses on other matters such as patient finances and happens to increase the rewards for biomedical innovation by embiggening the market, no one mentions it. But when Congress, focusing on patient finances, tries to rein in prices and thus decrease prices for industry, drugmakers scream bloody murder and claim that the engines of progress will grind to a halt. This legislative dynamic is off-kilter. It demands understanding and options for fixes. Sachs provides both.
Sachs grounds her analysis in a rich description of four keystone pieces of legislation that changed the landscape of biomedical innovation: the Orphan Drug Act, the Hatch-Waxman Act (occasionally known by its formal title, the Drug Price Competition and Patent Term Restoration Act of 1984), the Affordable Care Act, and the enactment of Medicare Part D (covering outpatient prescription drugs). Each of these Acts created substantial incentives for innovation. More obviously, the Orphan Drug Act and Hatch-Waxman Act each created new forms of exclusivity for drugs. Less obviously, the Affordable Care Act and Medicare Part D vastly expanded insurance coverage for drugs, including for the poor (ACA) and the elderly (Medicare Part D). This increased market size, prompting innovation in the relevant areas. As Sachs shows through meticulous legislative history, Congress talked lots about incentives for the first two, and essentially not at all for the second two.
Aha, you say! Maybe it’s just that Congress doesn’t understand that bigger markets create more incentives to develop products. But no—when Congress considers shrinking markets to protect patient pocketbooks, pharma promptly prognosticates plummeting productivity. And Congress listens. In fact, the Congressional Budget Office’s analyses of drug-pricing-cut proposals explicitly include estimates of how many drugs won’t be invented. So Congress makes biomedical policy accidentally, but only in one direction, when it increases incentives.
That descriptive insight is a valuable contribution on its own; Sachs has slogged through the legislative history so we don’t have to to understand why the ACA’s innovation policy implications remain underspecified. But what are we to make of all this? Sachs makes three incisive points.
First, if Congress has been making innovation policy accidentally, that policy probably demands pretty close scrutiny. For instance, did Congress really mean by changing insurance policy to increase incentives for new drugs for seniors, regardless of whether those drugs needed additional incentives? It doesn’t seem that they did, and if they did, that seems like a poor call and fodder for scholarly attention, especially when accidental policy results in innovation inequities.
Second, and related, the accidental and asymmetric nature of innovation policy means we should be especially cautious about baseline assumptions. If the status quo is at least in part an accident, there’s no particular reason to think we’ve magically wandered into the right answer. Incentives might be too low, or too high, or just misaligned to the type of innovation we really want. That last possibility seems awfully likely, based on separate work by Sachs, Hemel and Ouellette, Eisenberg, and others.
Third (and frankly the kind of insight that I’m glad someone like Sachs can share because it requires deep Congressional weediness), Congressional dynamics are just poorly set up to get the full innovation picture. Who would think that Congress misses big chunks of innovation policy because different committees have jurisdiction over different acts and see different bits of the elephant? Sachs. (The Judiciary Committee, for instance, sees patent laws, but not most health laws; Ways & Means sees Medicare but not intellectual property or FDA. For more, see…Sachs!)
The piece closes with institutional suggestions to fix Congressional innovation myopia. The Congressional Budget Office could tackle the task (though it doesn’t make recommendations or consider impacts of prior legislation); the sadly-defunct Office of Technology Assessment could evaluate broad pictures (though it, well, doesn’t exist); or the lesser-known Medicare Payment Advisory Commission or Medicaid and CHIP Payment and Access Commission (MedPAC and MACPAC, respectively and delightfully) could weigh in (though their foci are narrower than biomedical innovation writ large). Each has their problem, but any could help. It’s hard to argue with the idea that Congress should better know what it’s doing, and Sachs has identified an apparently substantial hole in that knowledge.
The argument Sachs makes is a compelling one. Congress is making big biomedical innovation policy by accident. One response is for Congress (and others) to actually think it through both prospectively and retrospectively, whatever the institutional mechanism. Another, drawing on Sachs’s other work, is to consider more broadly how creative tools can shape innovation policy more precisely. One reason Congress is making innovation policy by accident is because insurance payment and drug development incentives are so closely connected, often automatically. But they needn’t be; Sachs has argued that delinking reimbursement from development (and particularly approval) could help better align incentives (Hemel and Ouellette, Masur and Buccafusco, and I have also made suggestions in this direction.) Accidents of policy can be fixed or avoided, more so now that Sachs has so clearly delineated the problem.
May 12, 2022 Ifeoma Ajunwa
Amanda Levendowski,
Resisting Face Surveillance with Copyright Law, 100
N. C. L. Rev. __ (forthcoming, 2022), available at
SSRN.
One prevailing feature of technological development is that it is not sui generis. Rather, new technologies often mirror or reflect societal anxieties and prejudices. This is true for surveillance technologies, including those used for facial recognition. Although the practice of facial recognition might be positioned as a type of convincing evidence useful for identifying an individual, the fact remains that racial and gender biases can limit its efficacy.. Scholars such as as Timnit Gebru and Joy Buolawmini have shown through empirical evidence that facial recognition systems, which are often trained on limited data, display stunningly biased inaccuracy. The two AI researchers reviewed the performance of facial analysis algorithms across four “intersectional subgroups” of males or females featuring lighter or darker skin. They made the startling discoveries that the algorithms performed better when determining the gender of men as opposed to women, and that, darker faces were most likely to be misidentified.
In her path-breaking article, Resisting Face Surveillance with Copyright Law, Professor Amanda Levendowski identifies these harms and others, and advocates for the proactive use of copyright infringement suits to curb the use of photographs as part of automated facial surveillance systems. First, Levendowski illustrates why the greater misidentification of darker faces by algorithmic systems is a problem of great concern. Levendowski shares the story of Robert Julian Borchak Williams who was placed under arrest in front of his home and in view of his family. A surveillance photograph had been used to algorithmically identify him.. However, once the photograph was compared to the actual person of Mr. Williams, it was obvious that he had been misidentified. The only explanation Mr. Williams got was, “The computer must have gotten it wrong.” The sad reality is that Williams’ case is not unique, there are many more stories of Black men being wrongfully arrested based on misidentification by AI systems. Given the glacial creep of federal legislation to regulate face surveillance, Levendowski advocates for turning to the copyright tools she believes we already have.
Facial recognition systems have proliferated in the past few years. For example, in 2020, an individual taking the Bar exam in New York related how he was directed to “sit directly in front of a lightning source such as a lamp” so the face recognition software could recognize him as present. I have written about and against the troubling use of facial recognition by automated hiring programs. Evan Sellinger and Woodrow Hartzog have written about the extensive use of facial surveillance in immigration and law enforcement and have called for a total ban. Although some jurisdictions in the United States have heeded the call to ban the use of facial recognition systems by law enforcement, many others have not, and there is currently no federal legislation banning or even regulating the use of facial recognition systems.
Resisting Face Surveillance with Copyright Law is innovative in its approach of deploying copyright law as a sword against the use of automated facial recognition. As Levendowski argues “Face Surveillance is animated by deep-rooted demographic and deployment biases that endanger marginalized communities and threaten the privacy of all.” Deploying copyright litigation to stem the use of facial recognition holds great potential for success because as Levendowski notes, corporations like Clearview AI are trawling selfies and profile pictures online to compose a gargantuan face-recognition database for law enforcement and other purposes. Levendowski notes that Clearview AI has copied about three billion photographs without the knowledge or consent of the copyright holders or even the authorization of the social media companies that host those photographs. Levendowski’s article is one answer to what can be done with the laws we have now to curtail the use of face surveillance.
Levendowski notes that one common defense of scraping — to invoke the First Amendment — would not be viable against copyright claims. Levendowski recounts the Court’s statement in Eldred v. Ashcroft that “copyright law contains built-in First Amendment Accommodations”“ which “strike a definitional balance between the First Amendment and copyright law by permitting free communication of facts while still protecting an author’s expression.” Thus, Levendowski concludes, copyright infringement lawsuits could serve as “a significant deterrent to face surveillance” particularly given the hair-raising statutory damages of $150,000 for each case of willful infringement.
However, as Levendowski notes, there are several hurdles to the successful use of a copyright infringement lawsuit against face surveillance. For one, there is the affirmative defense of fair use. Levendowski concedes that the Google v. Oracle decision in 2021, which concluded that Google made a fair use when it copied interface definitions from Java for use in Android, has changed the fair use landscape and may make it less likely for copyright infringement suits against face surveillance systems to prevail. Yet, as Levendowski explains. the use of profile pictures may still fall outside of fair use protections because they are more likely to fail the four-factor test. She argues that unlike search engines which fairly “use” works in order to point the public to them, facial recognition algorithms copy faces in order to identify faces. That is, the “heart” of the copied work — a person’s face — is the part that is copied by the face surveillance systems, and the use is less transformative than a search engine’s use. Levendowski also draws on recent case law to suggest that courts will be less likely to find the for-profit subscription model deployed by many facial recognition companies to be fair use, compared to the free-to-the-public model used by most search engines.
Levendowski deploys Google v. Oracle and other key fair use cases to assess each fair use factor. First, she notes that surveillance companies are not using the pictures for a new purpose, their reason for using the photographs are the same as profiles pictures: particularized identification. Yet, Levendowski argues, even absent a new purpose, such use may still be somewhat transformative, favoring face surveillance companies. She then also concludes that that the nature of the work is creative and that the use features the photographs’ faces—the “heart” of profile pictures, creating unfavorable outcomes for these companies under the middle two factors. Analyzing the final factor, Levendowski concludes that using these photographs harms the unique licensing market for profile pictures, and that this dictates a ruling against fair use.
All in all, although some might not agree with her fair use analysis, I find Levendowski’s approach to be an ingenious approach to lawyering in the digital age. If I have any reservations, it is whether this information might introduce a new tactic for face surveillance corporations —to purchase or license the copyrights in the photographs they use. Such a tactic would be facilitated by social media or other platforms that require users to give up the copyrights to any photos they post. This indicates that there might yet be more regulation needed to address face surveillance. But in the meantime, Levendowski’s lawyering represents a creative approach to the problem of face surveillance.
Cite as: Ifeoma Ajunwa,
Confronting Surveillance, JOTWELL
(May 12, 2022) (reviewing Amanda Levendowski,
Resisting Face Surveillance with Copyright Law, 100
N. C. L. Rev. __ (forthcoming, 2022), available at SSRN),
https://cyber.jotwell.com/confronting-surveillance/.
Apr 7, 2022 Mireille Hildebrandt
In their brief and astute article Is That Your Final Decision? Multi-stage Profiling, Selective Effects, and Article 22 of the GDPR, Reuben Binns and Michael Veale discuss the arduous issues of the EU GDPR’s prohibition of impactful automated decisions. The seemingly Delphic article 22.1 of the GDPR provides data subjects with a right not to be subject to solely automated decisions with legal effect or similarly significant effect. As the authors indicate, similar default prohibitions (of algorithmic decision-making) can be found in many other jurisdictions, raising similar concerns. The article’s relevance for data protection law sits mainly in their incisive discussion of how multi-level decision-making fares under such prohibitions and what ambiguities affect the law’s effectiveness. The authors convincingly argue that there is a disconnect between the potential impact of ‘upstream’ automation on fundamental rights and freedoms and the scope of article 22. While doing so, they lay out the groundwork for a more future-proof legal framework regarding automated decision-making and decision-support.
The European Data Protection Board (EDPB), which advises on the interpretation of the GDPR, has determined that the ‘right not to be subject to’ impactful automated decisions must be understood as a default prohibition that does not depend on data subjects invoking their right. Data controllers (those who determine purpose and means of the processing of personal data) must abide by the prohibition unless one of three exceptions apply. These concern (1) the necessity to engage such decision-making for ‘entering into, or performance of, a contract between the data subject and a data controller’, (2) authorization by ‘Union or Member State law to which the controller is subject and which also lays down suitable measures to safeguard the data subject’s rights and freedoms and legitimate interests’ or (3) ‘explicit consent’ for the relevant automated decision-making.
Binns and Veale remind us that irrespective of whether automated decisions fall within the scope of article 22, insofar as they entail the processing of personal data, the GDPR’s data protection principles, transparency obligations and the requirement of a legal basis will apply. However, automated decisions are often made based on patterns or profiles that do not constitute personal data, precisely because they are meant to apply to a number of individuals who share certain (often behavioral) characteristics. Article 22 seeks to address the gap between data protection and the application of non-personal profiles, both where such profiles have been mined from other people’s personal data and where they are applied to individuals singled out because they ‘fit’ a statistical pattern that in itself is not personal data.
Once a decision is qualified as an article 22 decision, a series of dedicated safeguards is put in place, demanding human invention, some form of explanation and an even more stringent prohibition on decisions based on article 9 “sensitive” data (‘revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation’).
The authors are interested in the salient question of how different layers of automation create a disconnect between, on the one hand, the impact on the fundamental rights and freedoms of those targeted and, on the other hand, the protection offered by article 22. For instance, algorithmically inferred dynamic pricing (or ‘willingness to pay’) may be used to inform human decisions on insurance, housing, credit and recruitment. However, it escapes the GDPR’s protection against automated decisions because humans make the final decision. Considering ‘automation bias’, the presorting that takes place in the largely invisible backend systems may disenfranchise those targeted from the kind of human judgement and effective contestability that article 22 calls for. (See recently Margot E. Kaminski & Jennifer M. Urban, The Right to Contest AI.) The ensuing gap in legal protection is key to the Schufa case that is now pending before the Court of Justice of the European Union, raising the question of whether a credit risk score decided by the scoring algorithm of a credit information agency that is used by an insurance company, in itself qualifies as an automated decision (case C-634/21).
The authors distinguish five types of ‘distinct (although in practice, likely interrelated) challenges and complications for the scope of article 22. The first (1) is that adding human input at the level of all data subjects, which affects whether article 22 applies, can still leave a subset of data subjects not protected by that human input. The second (2) is the GDPR’s lack of clarity on ‘where to locate the decision itself.’ The third challenge (3) is whether the prohibition concerns potential or only ‘realised’ impact. The fourth (4) is the likelihood that largely invisible automated backend systems have a major impact irrespective of the human input that is available on the frontend. And the fifth (5) and perhaps most significant challenge is the GDPR’s focus on only the final decision in a chain of relevant decisions, which ignores the impact of prior automated decisions on the choice architecture of those making the final decision. This is the “multi-stage” profiling the authors reference in their title.
The abstruse wordings of article 22, probably due to compromises made during the legislative process, may inadvertently reduce or obliterate what the European Court of Human Rights would call the ‘practical and effective’ protection that article 22 nevertheless aims to provide. The merit of the points made by Binns and Veale is their resolute escape from the usual distractions that turn discussions of article 22 into a rabbit hole of fruitless speculations, for instance on whether there is a right to explanation and what this could mean in the case of opaque algorithmic decision-making and on whether the explanations are due before decisions are made or only after. As they explain, all this will depend on the circumstances and should be decided in light of the kind of protection the GDPR aims to provide (notably enhancing both control over one’s personal data and accountability of data controllers).
Binns and Veale’s precise and incisive assessment of the complexities of upstream automation and the potential impact on those targeted should be taken into account by the upcoming legislative frameworks for AI and by courts and regulators deciding relevant cases. In the US we can think of the Federal Trade Commission’s mandate and the National Artificial Intelligence Initiative Act of 2020. Binns and Veale remind us of the gaps that will occur in practical and effective legal protection if AI legislation restricts itself to the behavior of data-driven systems instead of incorporating decisions of deterministic decision-support systems, which will be the case if AI is defined such that the latter systems fall outside the scope of AI legislation. Both Veale and Binns are prolific writers, anyone interested in the underlying rationale of EU data protection law and the relevant technical background should keep a keen eye on their output.
