Rummaging Rebooted

Oct 1, 2025 Natalie RamAdd a Comment

Natalie Ram

Advances in digital surveillance technologies have posed difficult questions for Fourth Amendment doctrine. For instance, does the government need a warrant to install cameras on poles along a street to monitor who enters and exits homes? What if the government wants a list of all cell phones near a robbery scene at the time of the crime? Is the answer different if the government wants several days of data, but only about one person? What if the data comes from an app developer like Waze (or your flashlight app) or a smart home device like an Alexa, rather than a cell phone provider?

The Supreme Court has begun to address these issues in cases like Riley (barring warrantless cell phone searches during arrest) and Carpenter (requiring warrants for long-term cell phone location data). But as Andrew G. Ferguson argues in two recent articles—Digital Rummaging and Everything-Everywhere Searches—Fourth Amendment doctrine has nonetheless not kept pace with the scale of digital surveillance. In a turn to history that may prove particularly persuasive to constitutional originalists, Ferguson argues that the Founding generation’s objections to “rummaging” through general warrants provide an appropriate guiding principle for constraining surveillance in the digital age.

Ferguson warns that existing doctrinal focus on “reasonable expectations of privacy” and “trespass” may perversely encourage mass surveillance: “by searching everyone and everything at the same time, police can elide the traditional threshold search and seizure questions because it is not clear what expectations anyone has under such continuous surveillance or even when the search occurs.” As an antidote, Ferguson revives the Founding Era’s concern with “rummaging.” In Digital Rummaging, he introduces the “rummaging principle,” traces its historical roots, defines a “rummaging test” for courts, and applies that test to smart home data and long-term digital pole camera surveillance. In Everything-Everywhere Searches, he extends the rummaging principle to geofence warrants. Both articles merit close reading.

Ferguson traces the “rummaging principle” to the Founding generation’s deep mistrust of “government agents rummaging around homes, property, and papers.” Rooted in opposition to general warrants and writs of assistance, this principle has long served as a background constraint in existing Fourth Amendment doctrine. As the Supreme Court has observed, the Fourth Amendment was a direct response to these colonial-era abuses, “which allowed British officers to rummage through homes in an unrestrained search for evidence of criminal activity” (emphasis in Ferguson). Drawing on early sources, like Wilkes v. Wood and Entick v. Carrington, Ferguson shows how terms like “rummage,” “rifle,” and “ransack” captured the Founders’ fear of unchecked searches. While the Supreme Court has generally overlooked rummaging in defining what counts as a search, recent cases involving digital technology in policing, like Carpenter, have begun to resuscitate interest in this inquiry.

Building on this history, Ferguson distills the rummaging principle into a modern “rummaging test” constraining digital policing under the Fourth Amendment. He argues that courts should ask whether a contested search involves “(1) arbitrary enforcement of police power; (2) overreaching exploratory expansions of initially justified searches; (3) intrusions into constitutionally secured interests (e.g., homes, persons, papers, effects, location); or (4) exposure of private details as a form of political or social control.” These inquiries, Ferguson explains, align with the core harms that the Fourth Amendment was meant to prevent. Arbitrary enforcement occurs when unchecked police power leads to unreasonable interference with individuals or communities. Overreach happens when searches are too broad, such as using “probable cause pretext about one crime to search for other[s]” or sweeping innocent conduct or people up in investigations.

Intrusion refers to government efforts to access constitutionally protected spaces, people, or information. As Ferguson observes, protecting the home means safeguarding “the things that happen inside those four walls, not the walls themselves.” So too for people and, among other things, the information in their DNA. Finally, exposure involves the risk of revealing private information, recalling early privacy law concerns about the “privacies of life.” Government searches can create stigma that signals guilt to others and can become a powerful tool for social or political control.

Ferguson argues that this rummaging test can help determine both whether a “search” has occurred and whether a warrant, or other procedural or legal safeguards, makes that search reasonable. Ferguson also suggests that if a search causes significant enough rummaging harms, it may violate the Fourth Amendment even with a warrant.

The rummaging test clarifies decisions like Riley and Carpenter, which limited warrantless government conduct and “embraced—without necessarily acknowledging it—the principles behind the rummaging test.” It also casts doubt on older decisions, like Greenwood, in which the Supreme Court held that there is no Fourth Amendment protection for trash—an outcome Ferguson suggests fails to account for the harms of rummaging.

Turning to new forms of digital policing, Ferguson applies the rummaging test to smart home data and long-term pole cameras. Police typically use these tools based on mere hunches, hoping that rummaging through the data might turn up something useful. But if police can use these tools without a warrant, as prosecutors argue, it opens the door to arbitrary, overbroad, and deeply intrusive searches. Most of the information gathered would be “innocent, embarrassing, or irrelevant.” Smart home data may reveal details not otherwise “obtainable absent an entry into the home (if then),” while pole cameras could be deployed against disfavored individuals and entangle anyone with whom they socialize. Nonetheless, Ferguson suggests that with carefully crafted warrants—including “minimization requirements, time limits, or other considerations”—these tools might yet pass constitutional muster.

Finally, in Everything Everywhere Searches, Ferguson expands the rummaging test to digital surveillance that targets everyone’s data in hopes of generating a suspect list, or even just clues to the start of one. Ferguson focuses on geofencing, but similar mass queries arise in law enforcement use of consumer genetics data to generate leads by identifying genetic relatives of an unknown suspect, persistent aerial surveillance that records and stores data about everything that happens on city streets, facial recognition tools that can track or identify persons of interest, or tools like Shotspotter that are always listening and direct police to possible crime scenes where everyone present comes under suspicion. Ferguson identifies three characteristics these technologies: they are pervasive, capturing information in “widespread, comprehensive, and voluminous” ways; they are digital, enabling investigators to “search back in time, aggregate the data, and connect personal data points for new insights”; and they are indiscriminate, “collect[ing] information constantly against everyone, innocent, guilty, or anywhere in between.” These features often frustrate Fourth Amendment protection, especially when courts treat third-party data as beyond its scope.

Ferguson applies the rummaging test to geofence queries, where police ask companies like Google to identify all devices present in a specific area during a specific time. Police often use a geofence query when they have no suspect in mind, hoping that someone in the data will fit. Ferguson argues that warrantless geofence queries are classic rummaging—arbitrary, overbroad, and deeply intrusive. In this location data panopticon, “even just the potential of collection” could be chilling. Even with a warrant, problems remain, because of the inevitable involvement of private intermediaries.

Ferguson analyzes geofence warrants as currently conducted: authorized by courts, but mediated in a three-step process by Google. At Step One, Google scans its entire location database and returns anonymized data on all devices in the geofence—an overbroad search that sweeps in innocent people. Steps Two and Three narrow the pool and eventually identify individuals, but the initial dragnet remains constitutionally troubling. Ferguson warns that, even with these limits, “in terms of a grant of power, it is hard not to see the rhetorical parallels between geofence warrants and the general warrants that gave rise to the Fourth Amendment.” To be lawful, courts would need to impose a far more rigorous definition of particularity, and even then, such warrants might only “slightly alleviate” our concerns.

Ferguson’s expansive work on digital rummaging skillfully shows us one more way in which the doctrinal myopia on “expectations of privacy” or “trespass” can miss the real harms the Fourth Amendment was intended to prevent. His rummaging test offers a historically grounded lens for explaining the harms of big data searches that “invert the traditional investigative model.” This work invites fresh debate and legal challenges across a host of investigative methods, both well-established and new. Perhaps most controversially, Ferguson suggests that some surveillance practices may be so invasive that they should simply be off limits—warrant or not.

AI Disgorgement or AI Recalls: A Trip down Remedy Lane

Sep 3, 2025 Margot Kaminski1 comment

Margot Kaminski

In 2019 the Federal Trade Commission (FTC) created a new remedy in data privacy and AI law: algorithmic disgorgement, also known as model deletion. The FTC required that Cambridge Analytica “delete all Covered Information collected from consumers… and any information or work product, including any algorithms or equations, that originated, in whole or in part, from this Covered Information.” The idea behind model deletion is that companies should not be able to profit of models trained on wrongfully obtained personal data.

Algorithmic disgorgement has by now received its fair share of praise, including from FTC Commissioner Rebecca Kelly Slaughter, who called it “an innovative and promising remedy.” The remedy’s boosters, however, have largely lauded how algorithmic disgorgement/model deletion can mitigate data privacy and algorithmic governance laws’ struggles to identify, quantify, and deter legally cognizable harms.

Two excellent forthcoming articles—Daniel Wilf-Townsend’s The Deletion Remedy and Christina Lee’s Beyond Algorithmic Disgorgement: Remedying Algorithmic Harms—bring both more caution and more depth to the conversation. Both articles offer nuanced framings of algorithmic disgorgement as a remedy, and guiding thoughts on when and how it might most appropriately be deployed.

Wilf-Townsend acknowledges some of the benefits of model deletion (his preferred term, because he claims it really isn’t “disgorgement” at all in the traditional sense) while also criticizing its potentially disproportionate consequences. The article begins with a detailed account of the remedy’s rise. The Biden-era FTC, since Cambridge Analytica (2019), regularly deployed model deletion as a remedy: in its orders in Everalbum (2021), Weight Watchers (2022), Ring (2023), Edmodo (2023), Rite Aid (2024), and Avast (2024). (He and Lee cover much the same list of enforcement actions.) In litigation, however, model deletion has only barely entered the picture.

Wilf-Townsend calls the remedy “model deletion” because he argues that, despite use of the term “disgorgement” by former FTC Commissioner Chopra and FTC Commissioner Slaughter, the remedy really isn’t disgorgement. Fascinatingly, he argues that the fact that “algorithmic disgorgement” is a misnomer may preserve the remedy for the FTC’s use. The Supreme Court held in 2021 in AMG Capital Management that the FTC is not authorized under Section 13(b) to order retroactive monetary disgorgement (i.e., disgorgement of profits). But Wilf-Townsend points out that model deletion is prospective, not retrospective; and it’s not monetary, but behavioral. Thus, the FTC could still properly order model deletion as injunctive relief. In copyright law, the source of authority is clearer: 17 U.S.C. § 503 provides that courts “may order the destruction or other reasonable disposition of all . . . articles by means of which” unlawful copies “may be reproduced.”

Wilf-Townsend recognizes that model deletion can prevent ongoing harms caused by a model, such as the continued disclosure of private personal information or the direct reproduction of images in its training data. Model deletion also avoids the “difficulty of putting a dollar value on a harm” that is so prevalent in U.S. privacy law. Unlike damages, “model deletion… does not inherently need to be pegged to any sort of quantified harm.”

However, Wilf-Townsend is deeply concerned about the potential for throwing the baby out with the bathwater. He describes model deletion as it has been implemented thus far as amounting to a “no bad bytes” rule: if even some of the training data was obtained illegally, then the whole model goes down, regardless of where the model’s value originates, and regardless of potential social costs.

The problem per Wilf-Townsend is that model deletion as currently practiced does not require a showing that the unlawfully gathered or unlawfully processed data be the cause of a model’s value. He argues that for models trained on immense databases, like leading LLMs, “neither the law nor the logic of disgorgement would support the remedy of model deletion” because too little of the overall model’s function and value derives from what might be a relatively miniscule portion of its training data.

Wilf-Townsend closes by proposing “a test for determining whether to use model deletion in a given case.” That test assess how much of the value of a model is derived from unlawful data, which, in my view, would lead to valuation challenges that could undo some of the central benefits of resorting to algorithmic disgorgement in the first place.

Even if a model’s value is not primarily attributable to unlawful data, Wilf-Townsend suggests that model deletion might still be appropriate when considering the defendant’s degree of culpability, a balance of the hardships (similar to equity frameworks), and the availability of alternative remedies (including fine-tuning, unlearning, and filtering).

Where Wilf-Townsend’s article largely compares and contrasts model deletion with traditional monetary disgorgement, Christina Lee does further conceptual heavy lifting. Lee finds that what regulators have been calling “algorithmic disgorgement” (the term she uses throughout) in fact involves two different scenarios of harms and related remedies, tracing to two different underlying principles. This is fascinating work. Lee’s article does what the best articles do: sifts through some complex and sometimes nonintuitive sources to argue that bigger, hard-to-initially-see patterns are at play.

Lee begins by highlighting that the FTC’s use of the disgorgement remedy in Rite Aid marked a decided shift by ordering Ride Aid “to instruct any third parties that received the tainted data from Rite aid to delete… any models or algorithms trained on that data.” Importantly, in Rite Aid FTC went after the company not just for using unlawfully gathered data, but for using the facial recognition software unfairly.

This leads Lee to argue that the FTC has really been deploying not one but two distinct remedies: the first, data-based disgorgement that focuses on the provenance of the model (its unlawful training data); and the second, something more like a product recall, which focuses on the harms the use of a model is causing in the world. Lee convincingly argues that “[t]hey are two distinct remedies that happen to share the same mechanics.”

Lee argues that the data- and use-based remedies stem from two different principles: disgorgement and consumer protection. Where disgorgement attempts to undo wrongful profits stemming from lawless actions, consumer protection is driven by “the desire to avoid having in the market something that is likely to cause harms to a lot of people,” regardless of wrongdoing. They also address issues at different stages of the AI lifecycle. True disgorgement focuses on training data. In effect, consumer protection principles lead to a “disgorgement” that really is more like a postmarket AI recall.

The product recall work here is a must-read. As Lee notes, the EU AI Act empowers European authorities to order “recall” and “withdrawal” of AI systems. Product recalls in other fields stem from a product defect that is repeatedly observable during normal operation or reasonably foreseeable use. What is required is not a showing of scienter or even wrongful behavior, but “a pattern of hazardous defect.”

Lee explains that product recalls may be mandated by regulators, but are also often voluntary, or the result of regulatory nudging. Lee points out that unlike algorithmic disgorgement, recalls in practice occur as an escalating toolkit of remedies: from warning labels and minor repairs, to a requirement that a seller cease production and offer refunds. These escalating levels of recall, she argues, “balance the need to protect consumers from mass harm and the value of having a useful tool available, even if the tool poses some risks.” This is market-level consumer protection reasoning, consistent with the underlying principle she identifies.

The second half of Lee’s article shifts to a more practical critique of the remed[ies]. Lee draws on Katherine Lee et al and Jennifer Cobbe et al’s important work on AI supply chains to argue that the disgorgement remedy often misses the mark. This is both because many distinct actors may be involved in the creation and fine-tuning of an AI model, and because foundation models may serve as a sort of AI infrastructure (my term, not hers) on which other AI systems are built.

Two of Lee’s astute criticisms stem from these observations: that algorithmic disgorgement often has little impact on the actual wrongdoer, who might be elsewhere in the supply chain; and that algorithmic disgorgement may disproportionately affect innocent third parties, especially those using foundation models in different ways, for different purposes. Lee does, however, acknowledge that a consumer-protection-motivated disgorgement/recall might “be justified in certain circumstances…[i]f the offense is egregious, or the magnitude of the potential harm great.” But “in many instances, this will not be the case.”

I came away from these two great articles knowing a lot more about the substantive law and feeling able to situate it within helpful theoretical framings. I do think both, however, undersold the unique institutional story of the remedy. The FTC’s accelerated use of the disgorgement remedy occurred against the backdrop of its loss of monetary remedies in 2021. Both former Commissioner Chopra and Commissioner Slaughter appear to have served as norm entrepreneurs within the FTC, advocating for algorithmic disgorgement as deterrence. While each article covers their advocacy, neither makes a clear argument that the commissioners may have been constructing a replacement enforcement tool as other tools were taken away. Further, the institutional story entails looking at the FTC as a consumer protection agency. I would have liked to see both authors, but especially Lee, discuss the effect the FTC’s institutional values may have had on the development of and subsequent extension of the remedy.

These articles in my view represent crucial readings in large part because I suspect that unlike in the European Union, the U.S. approach to AI will largely end up being (or perhaps, already is?) primarily postmarket. As the backdrop to settlement negotiations, a motivator for creating AI safe-harbors through legislation, or the site of a significant pain point for AI companies, AI disgorgement represents a central regulatory tool in efforts to come.

Deepfakes Deconstructed

Jul 18, 2025 Jacob Noti-VictorAdd a Comment

Jacob Noti-Victor

With the rapid advancement of photorealistic generative AI technology, the problem of sexually explicit deepfakes has grown more urgent than ever. Thanks to widely available AI systems, users can now easily create images that appear to depict real people engaging in sexual acts. Not only have Taylor Swift and other celebrities been targeted, but deepfakes are also now alarmingly prevalent in American schools.

The government has already started to address the problem. At least 26 states now penalize the creation or distribution of nonconsensual sexually explicit deepfake imagery. And the federal Take It Down Act, which creates criminal penalties and a takedown regime for both real and AI-generated nonconsensual intimate imagery (NCII), was recently signed into law by President Trump. But, as Ben Sobel argues in his excellent (and award winning) new article, A Real Account of Deep Fakes, many of these bans have been passed without first articulating the precise harms posed by sexually explicit deepfakes, leaving the statutes open to free expression challenges. Sobel’s article aims to fill this gap. Through painstaking comparisons between deepfake bans and other areas of law that regulate deception, abuse, privacy invasions, and obscenity, the article crystallizes the normative arguments for deepfake regulation and the First Amendment stakes.

Beginning with a comprehensive survey of all recently passed or proposed state and federal laws, Sobel identifies several features common to many bans of sexually explicit deepfakes. In particular, these laws typically require the deepfake image to be a photorealistic depiction of an identifiable person, they prohibit distribution, and they do not require intent to deceive or harm. Most importantly, they do not allow the use of a disclaimer to avoid liability.

The fact that these bans hold distributors strictly liable, even if the deepfake images are clearly stated to be fictional, means that we cannot understand sexually explicit deepfakes as purely a defamation problem. Defamation requires a false statement that purports to be fact, meaning a disclaimer can generally be used to avoid liability. Sobel instead turns to privacy law to see if that offers a better fit. Building on recent work by Danielle Citron, Benjamin Zipursky, and John Goldberg, Sobel notes that the common law privacy torts are also mismatched with deepfake regulation. Some require the disclosure of true information, which deepfakes obviously are not. The tort of false light polices “offensive” distribution of false information but, like defamation, requires falsity. Privacy law does have ways of preventing the use of another’s likeness without permission, but these too fit deepfake bans unevenly. Claims under the right of publicity are generally limited to commercial uses. And “appropriation”—which Sobel treats as a cousin to the right of publicity that focuses specifically on dignitary harms—generally requires that the appropriation “advantage” the defendant.

Sobel ultimately concludes that deepfake bans are a kind of appropriation regime, but with a different normative core: “Today’s anti-deepfakes statutes redress the injury that appropriation redresses, subject. . . to the offensiveness limitation that appears in the false light tort.” That is, they focus on the “most offensive uses of identity—those that are (a) pornographic and (b) involve the manipulation of persons’ realistic visual likenesses rather than merely the invocation of their names.” The normative basis for deepfake regulation is thus “offensiveness” or “outrageousness,” of the kind that the law recognizes in a variety of areas, but one fraught with First Amendment uncertainty.

The article unpacks the normative and First Amendment stakes of this “offensive appropriation” rationale by turning to an unusual place: semiotic theory, and in particular the work of Charles Sanders Peirce. Semiotics is the study of signs—defined broadly as words, images, sounds, gestures—looking especially at how a sign’s meaning is created and communicated. Scholars have used semiotics in sophisticated ways to illuminate a variety of legal regimes, and Sobel’s work seeks to continue this tradition.

Semiotics distinguishes between two key types of signs: “indices” are signs that point to real-world phenomena (like a photograph) and “icons” are signs that resemble something but do not record reality (like a drawing). Deepfakes, as depictions that do not purport to document reality, are icons—they are closer to drawings than to something like documentary footage. This distinction is not merely semantic: recognizing that the law of deepfakes is fundamentally about the regulation of offensive icons yields interesting comparisons that illustrate the constitutional precariousness of these bans. Sobel’s comparisons include the prohibition on trademark dilution by tarnishment, bans on “morphed” child sexual abuse materials (materials where the image of a child is doctored to appear sexually explicit), and bans on flag and effigy destruction.

Rather than addressing each comparison, I will focus on one example that I think illustrates the value of Sobel’s turn to semiotics: written sexual fantasies. As cases like the notorious “cannibal cop” showcase, the First Amendment generally refuses to criminalize written sexual fantasies that involve real people, no matter how disturbing or obscene. But, as Sobel asks, what is the real difference between written sexual content involving a real person and a non-misleading deepfake? Neither are indices: they describe or depict identifiable people, but do not necessarily purport to document actual events, and both are offensive. Perhaps the visually realistic nature of a deepfake renders it so harmful that a categorical ban would not offend the First Amendment, similar to the ways courts have seemed to accept that morphed child sexual abuse materials (also categorizable as icons) are categorically outside the First Amendment.

Sobel does not claim to offer a doctrinal solution, but his analysis showcases that a blanket deepfake ban is, in essence, a content-based ban on expressive speech. States should be prepared to defend them as such, rather than hiding behind the inaccurate framing of defamation.

This analysis is subtle, and my one quibble is that Sobel could do a bit more to explicitly defend the need for semiotic analysis to make his main points, preemptively addressing those who might dismiss it as conceptual flair. More engagement with the rich literature on law and semiotics might help sway such skeptics. That said, I personally found the use of semiotic theory effective. The article rewards close reading, and Sobel is adept at threading complex social theory through many different areas of law.

Ultimately, Sobel’s work counsels us that even dire problems like sexually explicit deepfakes must be addressed judiciously to avoid undermining free expression and other constitutional protections. This is a lesson that we would be wise to apply to other problems posed by generative AI, which have led to a wave of new or proposed legislation. Many of these problems are serious, but their seriousness should not obviate the need for thoughtful analysis of AI’s precise harms and carefully tailored regulatory solutions.

The Problem of Insincere, Post-Hoc AI Explanations

Jun 20, 2025 Nicholson PriceAdd a Comment

Nicholson Price

AI is mysterious and important. It’s important because it’s showing up everywhere and doing lots of things. It’s mysterious because we very often don’t know how it works and why it comes to the conclusions it does. Whether AI should be important is hotly debated, but its mystery is widely regarded as a problem, particularly when AI is making inscrutable decisions that matter to people’s lives. And so there are widespread calls in law, policy, and scholarship for explainable AI—that is, ways to explain just why an AI system came to the conclusion it did. In The Algorithmic Explainability “Bait and Switch”, Boris Babic and Glenn Cohen add to the literature on explainable AI by clearly and convincingly arguing that explainable AI is “fool’s gold”—shiny and exciting on the surface, but not what we need, because it’s post hoc, insincere, tough to judge, and can’t be used to effectively guide actions.

So what is explainable AI, and why does it matter? Essentially, the problem is that it’s too hard to understand how AI makes decisions; they’re too complicated and don’t make sense, so they’re opaque to us. Explainable AI tries to use another, simpler algorithm to approximate a plausible reason the AI might have come to its conclusion; that explanation is typically specific to the conclusion being questioned. This happens after the initial system does its thing; it’s a post-hoc approximation, not a true accounting of why the initial system actually did what it did. Babic and Cohen illustrate this using an extended hypothetical admissions model for a hypothetical law school which shows the pitfalls and why they matter.

(As an aside, this bit demonstrates a real strength of the piece: its comprehensibility on complex topics. There’s a tension in law review articles: They need to speak to generalist readers (including the law students who do selection and editing, as well as scholars in adjacent fields), but they also need to move the ball forward for expert readers who are already in the conversation. It’s tough to do this well; typical approaches include neglecting one task or writing very long pieces with lots of detailed background to get the nonexpert up to speed. Both can be frustrating. Babic and Cohen smoothly walk this dual path, in part by using a sort of Choose-Your-Own-Adventure structure in the Background. ‘Here’s the math,’ they say, ‘but if you’d like, feel free to skip ahead to the intuitive example where we make it easy to understand.’)

Because AI explanations are simplified post-hoc approximations, they’ve got some real problems. They’re “insincere,” Babic and Cohen argue, in that they’re plausible reasons that the system might have used to make a particular decision (in the example, admitting a prospective student or not). But there’s no guarantee that they’re the actual reason. Indeed, there couldn’t be such a guarantee, because the whole point of post-hoc explanations is that they’re simple enough to be understood, when the whole reason we need post-hoc explanations in the first place is that the actual AI system being used isn’t simple enough to be understood. There’s a gap by definition. And so these answers aren’t sincere.

That post-hoc insincerity is a real problem for AI explanations for three big reasons. First, if an AI explanation doesn’t tell the actual reason for a decision, the affected party can’t know what to change to alter the outcome for next time (as an alternative, some have suggested systems for playing around with lots of possibilities to try to figure that out). It’s not an “action guiding” explanation if it can’t reliably guide action, something it’s often hoped explanations will do. If you’re trying to find out why your date to the movies is late and you only get a plausible explanation rather than the actual explanation, it’s hard to know whether to bail, buy a ticket for a later show, or get snacks because they’re on their way. (The article is spangled with delightful, intuitive examples that make tough concepts easier to understand, from Maverick and Goose piloting fighter jets to too-short dates to unethical test-ordering doctors; it’s a real strength.) More seriously, if someone gets denied parole and told a plausible reason that might or not be the real reason, it’s tough to know what to do to improve their chances for next time.

The second big problem with insincerity is trust. One touted benefit of explainability is that if people affected by AI systems understand their reasons, they’ll trust the AI systems (and the human systems in which they’re embedded) more. Transparency matters, and that includes knowing how decisions were reached. But if explanations are insincere and inaccurate, that’s likely to destroy trust in the system, not build it. This, Babic and Cohen point out, is especially likely because explainable AI comes up with different explanations for different individual decisions—and if the subjects of those decisions can share stories, they might find pretty quickly that they were given different decision rules.

Third and finally, it’s important to evaluate AI systems’ decision rules, because many rules aren’t OK. If a post-hoc, insincere explanation doesn’t reliably reflect the actual decision rule, it’s not a useful path to evaluate whether that rule is racist or sexist or otherwise unacceptable (which is disturbingly often the case).

The problems Babic and Cohen highlight matter because AI is incorporated into a broader range of contexts and decisions. When they wrote this piece in the hoary days of 2023, generative AI was still relatively new, and they focused accordingly on classification algorithms. But the problems of explanation remain, not only with those older systems but also with generative AI. Indeed, users can ask a chatbot why it said what it said. Trusting the answer is another matter. These issues aren’t going away.

So what’s to be done? There’s always the hope for a technological deus ex machina that makes all the black-boxes transparent and explicable; that’d be lovely but seems unlikely, at least in the near term, whether because it’s computationally very expensive to peer inside even simple black boxes or because some black box mechanics simply aren’t explicable. Instead, Babic and Cohen argue, we need to face up to the reality that explainable AI can’t really do all that’s asked of it. In some circumstances, that means we need to rely on interpretable AI or algorithms instead (simpler models we can actually understand); the Fair Credit Reporting Act takes this approach, for instance. Where procedural justice or democratic freedom are at stake, we truly need to understand why decisions are reached. In other contexts, we might be willing to sacrifice understanding in service of better performance; many medical AI systems might fall into this bucket. In any case, we should be clear-eyed about what we’re doing. With Babic and Cohen’s sharp and cogent explanation of explainability, that’s an easier task to undertake.

Centering the Vulnerable through Data Protection

May 22, 2025 Scott Skinner-ThompsonAdd a Comment

Scott Skinner-Thompson

For American lawyers, the concept of data protection can seem overly bureaucratic and even a bit obtuse. American legal scholars, in general, prefer to think in terms of privacy, with its manifold methods of potential protection of the liberal individual subject via tort causes of action, criminal law, consumer protection, and, occasionally some actual command and control regulation. In other words, the concept of data protection can—again, particularly for American audiences—seem question begging: protection of what data, whose data, and from whom? (Clearly the same questions can and are asked about privacy protections).

In his recent book, Professor Gianclaudio Malgieri explains why data protection laws matter. The GDPR isn’t an annoying consent regime for internet browsing, but can be mustered to protect people along several axes of vulnerability—including their demographics, yes, but also any power imbalance relative to the data controllers. The GDPR isn’t ideal for guarding against vulnerability because it lacks clear and explicit protections for the precarious and, according to Malgieri, new regimes must be imagined and implemented. But the book’s critically optimistic view helps us see how data protection can be used here and how to guard against vulnerability; in essence, as a form of harm reduction. It is a rigorous book that deftly applies often ethereal (but important) philosophical concepts to a turgid regulatory regime in order to unpack that regime’s anti-subordination potential.

How so? To begin, Malgieri explains while, on its face, the GDPR seems geared toward protecting an “average” data subject, there is room for consideration of contextual factors that might make the law more attentive to the needs of vulnerable subjects. Drawing from the work of Professor Martha Fineman and others, Malgieri recognizes that vulnerability is not a static concept tied to any specific demographic identities, but is a dynamic one that captures various kinds of power imbalances and intersectional identities. He then documents how European law makes room for the concept of a dynamic vulnerable subject in various contexts ranging from human rights to consumer protection. He believes there is support for incorporating this approach into the interpretation of the GDPR in part because of the GDPR’s solicitude for certain kinds of individuals, particularly children, and particular kinds of information, including the so-called special category data or sensitive data.

Assuming that is true, Malgieri explains how the GDPR can be interpreted to consider vulnerability both when evaluating whether data processors are complying with their duties as to those individuals and in determining whether individuals have the capacity to take advantage of the GDPR’s consent-and objection-based safeguards. In other words, there may be some hard and fast limits on what data can be processed with respect to vulnerable individuals. In particular, Malgieri sees potential for the data-protection impact assessments (DPIA) required by the GDPR as a fertile space where vulnerability concepts can be implemented with alacrity.

Make no mistake, Malgieri is clear-eyed that the GDPR is no magic wand for protecting vulnerable data subjects. And he recognizes both that his reading of the GDPR’s obligations with respect to vulnerability is aggressive (albeit textually strong), and that the GDPR could be amended to more explicitly capture the plastic concept of vulnerability without making it so flexible that it loses force and meaning. But Malgieri’s book does a truly commendable job of doing what lawyers ought to do: lawyer. It makes strong textual and normative arguments to advance the law toward justice and it does so in a methodical, disciplined, and yet accessible way. It’s a tremendous intervention for all those concerned about anti-subordination in the digital and physical spheres.

Products Liability as the Pragmatic First Step to A.I. Regulation

Apr 23, 2025 Ifeoma AjunwaAdd a Comment

Ifeoma Ajunwa

With her recent article, A Products Liability Framework for A.I., Professor Catherine Sharkey may have silenced at least some critics of artificial intelligence (A.I.) regulation. At the very least, the article stands as a sharp retort to anti-regulation advocates who often crow: “But how can we regulate A.I. when we don’t even yet know the full extent of what it can do or how it will be used?” Sharkey’s proposed regulatory framework, which eschews ex-ante pre-approval strategies in favor of post-market regulatory monitoring, may just be the answer to one of the critics’ favorite regulatory dodge.

Sharkey has the savoir faire to be afforded credence for any A.I. regulation proposal. As both an A.I./ML (machine learning) law and tort law scholar, what most stands out about Sharkey’s oeuvre is that she has gained enviable access to observe how A.I./ML systems are deployed in the government and has deployed her admirable analytical skills in dissecting those workings. For example, in Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies, Sharkey (along with other scholars), conducted a rigorous canvass of A.I. use at 142 federal departments, agencies, and sub-agencies. Sharkey et al’s work in Government by Algorithm has been an inspiration for other scholars taking up the mantle to advocate for guardrails to automated governance.

I found reading A Products Liability Framework for A.I. to be similarly highly generative in my thinking of regulatory legal mechanisms, and I believe this article will become canonical for A.I. legal scholars grappling with the challenges of regulating emerging A.I. technologies. First, the Article notes the peculiar regulatory challenges posed by A.I./ML given their adaptive nature. Sharkey observes, “[c]ritics suggest that regulating A.I./ML demands a unique regulatory approach because, as A.I./ML technologies are sent out into the world and encounter new situations, they learn and change in real time.”

The first helpful contribution of the Article is that Sharkey handily demonstrates why A.I. technologies could be considered “products.” She seizes on the FDA’s stance for governing A.I./ML medical devices as products as a lodestar. Ultimately, she argues for a functional approach, advocating that A.I. technologies should be considered products due to their mass-market distribution and potential for widespread harm, since these are the same underlying public policy concerns of products liability law. Sharkey contends that classifying A.I. as a product ensures that liability frameworks remain effective in protecting consumers.

After establishing that A.I. should be considered a product, Sharkey’s article is built around the idea that the uncertainty produced by the ever-changing nature of A.I. development and use is neither peculiar to that technology, nor an insurmountable challenge to regulation. Rather, other emerging technologies have presented the same uncertainty in their nascent years and those regulatory challenges were still governable.

To Sharkey, the key to those early governance problems was products liability. As she notes in this Article and in previous writings: “Products liability…is a microcosm of how the common law evolves over time to respond to new societal risks—historically, those posed by the automobile, mass-produced goods, digital e-commerce…” Therefore, for Sharkey, it follows that product liability legal frameworks may also work well for regulating emerging technologies like A.I. She argues that products liability law affords legal mechanisms, including: an information-forcing function for safety-related information while more proactive regulatory frameworks are being developed, a liability insurance regime, and the added efficiency of applying the cheapest cost avoider theory.

Sharkey argues: “We can draw lessons from historical examples where society faced new and uncertain risks to demonstrate that, even when risks are uncertain or not entirely understood, tort liability can serve an information-production function during a “transitional period” before an ex ante regulatory scheme is in place.” Second, Sharkey notes the role of liability insurance, especially to produce information and enforce standards to mitigate or prevent harms from A.I. She writes, “Liability insurers can aggregate risk-related information obtained about the expanding universe of policyholders as part of the process of underwriting and premium-setting.” Third, Sharkey believes that the “cheapest cost avoider” theory serves as an effective deterrence. As applied to A.I., the “cheapest cost avoider” framework is less concerned with A.I.’s “Black Box” problem because it is concerned only with reducing the societal cost of accidents. According to Sharkey, “Instead of attempting to attribute each A.I. output to a single party, courts would focus on whether the interactive user or the A.I. developer is in the best position to mitigate or prevent harms.”

The cheapest costs avoider rationale is firmly grounded in the torts literature and was proposed by Professor Guido Calabresi in his groundbreaking book, The Cost of Accidents. Yet Calabresi and Smith also provide something of a warning: “But what is “cheap” and what is “costly” itself derives from the tastes and values of society, which can be influenced by the current set of civil wrongs. This reverse link, which is sometimes missed, may well represent the future of tort law.” This quote demonstrates how what is allowed by law (i.e., the parameters of civil wrongs) may then come to determine the values of society, i.e., what is socially acceptable. In the context of A.I. regulation, we should be attentive to how products liability, as a method of regulation, may come to define what A.I. technologies corporations will develop for society.

Thus, as admirable as I find Sharkey’s intellectually nimble analyses comparing emerging A.I. technologies to other prior emerging technologies regulated by products liability, I must note one concern. Sharkey argues that her proposal aims to balance innovation with consumer protection. I understand her instinct. However, some scholars take issue with regulation being posited as adversarial to innovation and consider the foregrounding of innovation in the governance conversation to be a regulatory dodge in disguise. Given that, as Calabresi and Smith note, what is cheap and what is costly depends on the “tastes of society,” we should question what an innovation-centric paradigm means for A.I. regulation. As Andrew Selbst concluded in Negligence and AI’s Human Users, “[w]here society decides that A.I. is too beneficial to set aside, we will likely need a new regulatory paradigm to compensate the victims of A.I.’s use.”

Is product liability law malleable enough to identify and quantify the harm to all victims of A.I. use? Tort law at its base relies on quantification. There is no recovery for damages if a plaintiff cannot quantify the harm. Thus, products liability may not compensate for reputation and representational harms which are often future or speculative in nature. Consider that privacy law scholars are still valiantly attempting to quantify the harms of privacy violations and that A.I. technologies introduce new opportunities for privacy violations. Even if the harm can be quantified, given that A.I. is being developed by multinational corporations with a deep bench of lawyers and even deeper pockets, is the financial asymmetry too great for any consumer of A.I. to be protected by products liability? My deep worry here is that although Sharkey has presented a noble effort to start to corral the dangers of A.I. innovation, A.I. developers may seize on it as carte blanche to push what they consider A.I. innovation at high cost to human life – what I would term a “break things and pay damages later” approach.

But what is the alternative? I underscore here that Sharkey has positioned her proposed legal framework as a stopgap rather than the end goal of A.I. regulation. I find her proposal then to be a highly creative and ultimately useful temporary solution. Turning to the question of what should be the ultimate objective of regulation, I would argue for a reimagining of our legal principles vis à vis the responsibility of corporations. To be more precise, effective regulation of A.I. technologies will hinge on finding a definitive answer to a longstanding jurisprudence question: How can we expect corporations to evince true corporate responsibility towards society at large, while holding on to the shareholder primacy principle?

EU ICT: A Tale of Regulation, Innovation (?), Causation (?) and Frustration (!)

Mar 26, 2025 Tal ZarskyAdd a Comment

Tal Zarsky

There seems to be a budding consensus among tech pundits and stakeholders: The EU has solidified its role as a leader in one ICT sector—regulation. EU regulation is a growing industry in itself. However, such regulation may not necessarily be beneficial for business and technological progress. Professor Bradford, a leading expert on EU law and its international influence, agrees with the first two statements, but not necessarily with the third. She challenges (and ultimately rejects) the intuitive argument that excessive ICT regulation is responsible for the EU’s innovation lag in this sector. In making her claim, she maps out the many impediments to ICT innovation in Europe, identifying numerous factors beyond the content of regulation – such as its complexity, as well as underdeveloped capital markets, unfitting insolvency laws and the inability to attract and retain talent. Or, to paraphrase J.F.K.: Bradford explains that the EU’s ICT innovation failure has many fathers. Bradford thus argues that the link between regulation and the lack of innovation is weak and that there is no real lesson here for U.S. regulators and lawmakers contemplating tech-related policy.

To illustrate the weak connection between innovation and regulation, Bradford begins the article by outlining the U.S.’s centrality in the ICT sector. She highlights the dominant brands like Google, Meta, Microsoft, Amazon, and Apple that shape contemporary life and discourse, as well as the extraordinary wealth these firms have amassed. The article then examines the U.S.’s tech-friendly regulatory environment, particularly the relative immunity provided by Section 230 of the Communications Decency Act (as part of broader notion of promoting free speech) and the absence of comprehensive federal privacy legislation.

Bradford argues that this regulatory landscape was shaped by persistent lobbying efforts and an overarching U.S. policy commitment to “free market ideals.” She then systematically reviews key European laws and regulations affecting the tech industry— the General Data Protection Regulation (GDPR), Digital Markets Act (DMA), and Digital Services Act (DSA), among others. Beyond providing a thorough analysis of lobbying positions on these issues (and their influence on political discourse), Bradford acknowledges scholarly perspectives (including my own) that have suggested a possible connection between the EU/U.S. regulatory divide and the U.S.’s undisputed leadership in the tech market. The article subsequently engages with broader scholarly discussions on the relationship between innovation and regulation, while striving to prove that heavy regulation in the EU is not the key reason for the continent’s innovative lag.

Bradford then delves into specific regulatory domains and their relationship with innovation: privacy, antitrust, and AI. I will set aside the antitrust discussion, as Bradford’s argument that antitrust enforcement is crucial for innovation is fairly well-established. The relationship between privacy regulation and innovation, however, is more complex. Here, Bradford focuses on how the GDPR may hinder innovation by imposing high compliance costs. But she also explains how privacy laws “have the potential to alter innovation pathways” in different directions, some of which leading to the introduction of privacy-enhancing tools. At the same time, they might also “increase social innovation” (i.e., enhance social welfare as opposed to mere corporate wealth). She applies a similar analysis to the tensions between AI innovation and AI regulation. Subsequent research might consider linking these discussions of technological and social innovation and examine the impact of privacy laws on AI innovation (for a recent exploration of this connection, see Dan Solove’s recent article). The effect of the EU’s privacy laws (in the form of data usage restrictions) on AI development within the continent is likely to be significant.

If tech-related regulation is not the primary reason for the EU’s ICT lag, what is? Bradford identifies several alternative explanations for Europe’s limited technological leadership. Yet before doing so, the analysis presents three particularly insightful arguments, all aimed at weakening the assumed correlation between regulatory intensity and innovation constraints. First, she argues that EU regulation was not significantly different from that of the U.S. until 2010. Yet even during this period, Europe failed to produce ICT leaders, suggesting that other factors are at play. Second, she emphasizes that EU regulation serves dual objectives—protecting rights and fostering the internal EU market—and that the latter goal should, in principle, promote innovation and counterbalance regulatory burdens. Third, she points out that GDPR enforcement has predominantly targeted U.S.-based companies, with little evidence that innovation in these firms has been stifled as a result. While each of these claims can be countered, they are intriguing as part of the ongoing debate.

The final section of the paper examines structural impediments to innovation in the European tech sector, including:

1. Regulatory complexity rather than regulatory stringency – Bradford notes that the key issue is not necessarily the severity of regulation but rather the complexity arising from fragmentation across member states and the absence of a true “Digital Single Market.”
2. Limited capital markets – Europe faces a shortage of venture capital funding for startups, as well as limited government investment in military-driven technological innovation (more on this, later).
3. Punitive insolvency laws and a risk-averse culture – European legal frameworks discourage entrepreneurial risk-taking.
4. Inability to attract and retain global talent – This challenge is compounded by higher salaries in the U.S. and stronger institutional connections between academia and industry.

Summarizing these points, Bradford states: “Identifying these alternative explanations does not support an argument that all European tech regulation would enhance welfare and that digital regulations could never adversely affect innovation and slow down technological progress…”. She ultimately calls for a more nuanced discussion on the benefits and drawbacks of tech regulation.

This is a particularly timely article and discussion. Since its (very recent) publication, much has already changed. In the U.S., the new administration appears to be shifting even further toward a pro-business trajectory, particularly regardingthe tech industry. Thus, substantial regulation of the tech sector seems unlikely, except for some rules prohibiting certain forms of content moderation and censorship. Meanwhile, Bradford’s discussion of U.S.-China tensions has become increasingly relevant given the recent success of Chinese AI ventures like DeepSeek. Bradford’s intervention serves as an important reminder that competition in the ICT space is also coming from China. Consequently, the importance of the discussion Bradford chooses to promote has significantly grown. I recommend keeping the potential AI- and tech-related competition coming out of China constantly in mind when considering the arguments noted above. I also call attention to Bradford’s last book, “Digital Empires,” as an important source to acknowledge when reviewing the central approaches to regulating the digital economy, and the differences between them.

Bradford’s work is currently being supplemented by a growing body of scholarship examining the impact of the GDPR on innovation. Even if one accepts Bradford’s argument that the EU’s lack of ICT leadership is not directly attributable to privacy regulation, the GDPR’s enactment presents a valuable opportunity for empirical analysis: a natural experiment (though the absence of a “control group” limits the robustness of such findings). The GDPR serves as an instance in which an additional regulatory burden was introduced, allowing researchers to compare innovation trends before and after its implementation. Scholars have already started researching this question, but their findings remain inconclusive: some studies indicate lower levels of innovation in cutting-edge projects but increased innovation in more compliance-oriented sectors. Yet others showed a “variety of partly countervailing effects”.(see also a review of these discussions in Esra Damir’s dissertation, Ch. 3 and 7).

Looking forward, the introduction of strict EU regulations will allow for closer examination of the potential causal links between regulation and innovation (or the absence thereof). These developments may also facilitate a more precise assessment of the actual costs borne by EU citizens because of stricter data protection and information privacy laws, as well as the broader economic implications of such policies. Bradford’s paper provides a great start, with lots of thoughtful ideas and important facts for those striving to gain a deeper understanding of the reasons for the EU’s ICT innovation standing, including those related to various regulatory realms. Following its lead, questions regarding the connection between innovation and regulation will surely generate additional academic interest in the years to come.

Federal Civil Litigation in the Age of Generative AI

Mar 3, 2025 Stacy-Ann ElvyAdd a Comment

Stacy-Ann Elvy

Increasingly, attorneys use various generative artificial intelligence (AI) tools in the practice of law. These tools purport to provide targeted answers to specific legal questions and they can be used to facilitate review and drafting of legal documents as well as aid in due diligence assignments, along with various other legal tasks. In response to the rapid rise of generative AI tools in the legal profession, state bar associations have published recommendations on the issue. For instance, in 2023, the California State Bar Association issued practical guidance to attorneys on generative AI in the legal profession. Florida followed suit by issuing an advisory opinion on the topic. Similarly, the American Bar Associationalso released a formal opinion on generative AI tools in 2024.

In her article, Rule 11 is No Match for Generative AI, Professor Jessica R. Gunder offers an impressive contribution to both the law-and-technology and civil procedure fields by exposing the limits of Federal Rule of Civil Procedure 11 in addressing “fictitious cases and false statements of law” that arise from attorneys’ use of generative AI. Gunder convincingly argues that although courts have used Rule 11 to sanction attorneys who fail to conduct sufficient legal research, Rule 11 cannot adequately regulate this behavior in the generative AI context. She goes on to contend that Rule 11’s inadequacies have likely led a growing number of courts to issue standing orders to directly address attorneys’ misuse of generative AI in legal proceedings.

Gunder begins the article with a valuable description of the features associated with generative AI in the legal profession. She documents and critiques well-known cases in which attorneys improperly used generative AI tools. Gunder offers possible explanations for lawyers’ unprofessional use of generative AI, including their failure to understand the technology and “evaluate the work product” produced by the technology.

Gunder then turns her attention to Rule 11. After providing a brief overview of Rule 11’s history, scope, and objective, Gunder contends that attorneys and litigants can violate Rule 11 by filing legal documents that contain an inaccurate representation of law or that “do[] not contain key cases.” However, she argues that even before generative AI, courts already encounter significant hurdles in attempting to determine whether to impose sanctions for failure to perform sufficient legal research, including problems with identifying “how much research is enough?” Due to these difficulties, she posits that courts are reluctant to sanction attorneys for inadequate research. Given attorneys’ ethical obligations, courts are more likely to impose sanctions when there is “an intentional failure to disclose controlling legal authority” or if the conduct “is repeated, particularly after a court has informed the attorney of their error” or “involves misrepresenting or changing the holding of a case.”

Gunder goes on to argue that due to sanction requirements, Rule 11 will largely be ineffective when a litigant or attorney erroneously relies on generative AI and submits legal documents to a federal district court that contain “fictitious cases and false statements of law.” Gunder posits that Rule 11 is not intended to cover all bad faith conduct in a lawsuit and that the rule “cannot be used to sanction oral misrepresentation and testimony.” She argues that cases involving litigant or attorney misuse of generative AI often stem from “lack of knowledge of how generative AI works and its propensity to hallucinate” and while such conduct is perhaps negligent, it does not rise to the level of “contempt or subjective bad faith” for purposes of imposing Rule 11 sanctions.

The well-written article concludes with an examination of standing orders dealing with attorney misuse of generative AI and recommendations for courts moving forward. Gunder argues that standing orders may encourage litigants to refrain from filing legal documents in court that contain inaccurate statements of law or fabricated cases. Moreover, “they may make it easier for a court to find that a litigant violated Rule 11 and impose sanctions.” Despite these potential benefits, she contends that poorly drafted standing orders may discourage litigants and attorneys from adopting and implementing new technology and that “a patchwork of standing orders” issued by different courts may lead to inconsistencies.

Gunder suggests that courts should effectively balance the benefits and risks she associates with generative AI standing orders. She also posits that courts should be reticent to adopt “an anti-technology tone” in standing orders to avoid deterring parties from adopting generative AI and the appearance of “judicial bias.” She advocates for the use of the federal district court local rules process authorized by Federal Rule of Civil Procedure 83 to remedy concerns associated with inconsistent standing orders. Gunder’s insightful description of the current use of generative AI in civil litigation in federal courts should be of particular interest to courts, practitioners and scholars in both the law-and-technology and civil procedure fields.

State Surveillance, But From the Ground Up

Feb 6, 2025 Ari WaldmanAdd a Comment

Ari Waldman

Over a year before the Supreme Court’s conservative supermajority overturned Roe v. Wade, the Texas legislature passed SB 8, which banned all abortions after six weeks. At the time, fetal heartbeat laws like SB 8 were invalid because Roe and its progeny prohibited the use of state power to prohibit access to abortion services pre-viability. So Republicans in the Texas legislature, supported by the work of anti-abortion movement lawyers, came up with a workaround. SB 8 deputized private citizens to surveil on the state’s behalf and authorized them to bring private civil lawsuits against anyone who provided or facilitated an abortion after six weeks. SB 8 is a perfect and heinous example of what Sarah Brayne, Sarah Lageson, and Karen Levy call “surveillance deputies.”

In Surveillance Deputies: When Ordinary People Surveil for the State, Brayne, Lageson, and Levy define surveillance deputies as “ordinary people us[ing] their labor and economic resources to engage in surveillance activities on behalf of the state.” From one perspective, surveillance deputies are paradigmatic of the engaged citizen: “If you see something, say something” is not, in this understanding, a McCarthyite or totalitarian slogan encouraging tattling and ratting on neighbors. Instead, it’s a message about what constitutes good citizenship. Good citizens speak up and keep everyone safe. From another perspective, however, surveillance deputies are decidedly sinister. The connection between speaking up and keeping everyone safe implies that those listening to surveillance deputies have the best interests of citizens in mind. That is far from a sure thing. Surveillance deputies expand the power of the state and sometimes do so for the mere sociopathic reward of seeing someone else harmed.

At a minimum, surveillance deputies are a conundrum. How are we to understand the people who surveil and the institutional alliance between state power, the surveillance-industrial complex, and ordinary citizens? Brayne, Lageson, and Levy, who contributed equally to this outstanding and insightful article, propose four hypotheses for describing the functions and implications of surveillance deputization: interest convergence, legal institutionalization, technological mediation, and social stratification. Let’s break those down.

By interest convergence, the authors mean that surveillance deputization works best when states and citizens have aligned interests and benefits. For instance, under SB 8, someone could report an abortion provider in Texas for the chance to win $10,000 per incident, or because they hate abortion, or because they have a grudge against a doctor. For private deputies and the state, which wanted to end abortion in Texas, it was a win-win (a lose-lose for just about everyone else, but that’s a different JOT).

Surveillance deputization can also be catalyzed by law and its loopholes. Fourth Amendment case law holds that deputizing an individual to do surveillance work means that the state can avoid many of the constraints typically imposed on state surveillance. Given the access we have to vast amounts of surveillance content, this kind of exception to the Fourth Amendment’s warrant requirement may soon make the provision practically meaningless.

In addition to aligned interests and legal loopholes, profit lies at the foundation of much surveillance deputization. Big tech companies like Amazon and myriad small startups develop and market surveillance technologies to capitalize on people’s fears: fears about the “other”, about what will happen to their children, about anything. The information industry has pushed the notion that ordinary citizens should be monitoring everything, keeping an eye on what’s going on outside their door, by creating the very tools that give people those capabilities. Then they can sell advertisements on their surveillance apps. Having molded citizens into both consumers and avid spies, industry takes the resulting massive treasure trove of data and enters into lucrative contracts with state bureaucracies of violence to provide that data for the state’s use. As one former employee of the company that makes the Citizen App admitted, “The whole idea behind [the Citizen app subscription service] is that you could convince people to pay for the product once you’ve gotten them to the highest point of anxiety you can possibly get them to.” Create surveillance, stoke fear, cash in.

Finally, surveillance deputization may increase or disrupt social inequalities. The former is typified by the Victims of Immigrant Crime Engagement (VOICE) hotline. VOICE was set up by Donald Trump to let people call in and report what they thought were crimes being committed by immigrants. Since no one knows anyone else’s immigration status from afar, this hotline was basically an opportunity to report people of color to ICE. At the same time, technologically mediated surveillance allows citizens to surveil the state and its agents when they engage in racist or discriminatory behavior. In fact, the deputization of surveillance in a technologically driven world opens up a natural path for resistance: mess with the tech. Instead of leaving SB 8’s reporting website to anti-abortion fanatics, someone created a bot that submitted false reports every 10 seconds, overloading the platform and undermining the entire reporting structure. A similar thing happened to VOICE.

Surveillance Deputies highlights underappreciated aspects of the deeply symbiotic relationships between technology and state power. The authors give many examples—some good, some bad, and some ugly—of surveillance deputization beyond SB 8. AMBER alerts engage communities to assist in searching for missing children. The Amazon Ring doorbell camera may be user-installed, and the associated Neighbors app gives individual customers the chance to upload videos of what they see as “suspicious” activity, but the state regularly accesses Neighbor app data. And, of course, there’s VOICE, a way to turn every person who doesn’t look like you, you’re scared of, or you don’t like into an alleged criminal.

But the litany of examples raises one lingering question: When are we as citizens not surveillance deputies? Our labor and economic resources power almost every tool that is data driven: Google Maps, social media, targeted advertisements, and more. As it functions today, much digital infrastructure would collapse if users stopped contributing their own labor (for free) to multibillion dollar technology companies. Perhaps it is time for our own brand of resistance.

Can The Buyer Beware? State Procurement of AI Systems

Dec 11, 2024 Orla LynskeyAdd a Comment

Orla Lynskey

The role of private digital infrastructure providers in shaping the exercise of civil liberties in the digital sphere, and the role the law plays in facilitating this power, have been the subject of debate in recent years. Relatively less attention has been paid to the impact these ‘new governors’ have on the delivery of public services. As the State becomes increasingly dependent on privately provided AI systems, there is a real risk that public values (such as participation, transparency, and accountability) will be weakened. Historically, procurement rules have been used to ensure that public-private partnerships align to public objectives and values. Many lawyers, myself included, therefore surmise that when the State buys AI systems to assist with the delivery of public services, public procurement law will act as a constraint on the power granted to private operators by the arrangement.

In Responsibly Buying Artificial Intelligence: a ‘Regulatory Hallucination,’ Albert Sanchez-Graells clinically dispels such misplaced faith in procurement law, labelling it a ‘regulatory hallucination.’ Like AI hallucinations, this type of regulatory hallucination is ostensibly plausible but ultimately incorrect, leading to immediate tangible consequences (such as the mass harm that resulted from the Australian government’s wrongful demand that welfare recipients pay back benefits based on the Robodebt system). While Sanchez-Graells’ primary analytical focus is the UK, where under the National AI Strategy, public buyers are expected to ‘confidently and responsibly procure AI technologies for the benefit of citizens’, the logic of the argument applies also to other jurisdictions.

The main thrust of the article, and the book which further expands on the claims, is that the public buyer is badly placed to act as a public sector digital gatekeeper and self-regulator. According to UK Government’s AI policy, AI should conform to high level principles including fairness, accountability, contestability, and safety (amongst others). Responsible AI procurement therefore requires public buyers of AI to translate these substantive requirements into tractable contractual terms, which Sanchez-Graells terms AI ‘regulation by contract’. While there has been some positive experience in the UK of using procurement to achieve societal goals (such as environmental protection), this has not been an unmitigated success. More importantly, Sanchez-Graells illustrates how there are two assumptions underpinning the presumed effectiveness of regulation-by-contract that simply do not hold true in the digital context.

The first assumption is that AI regulation-by-contract can act as a two-sided gatekeeper disciplining the behaviour of both the tech provider and the public sector user of AI (for instance, a Welfare Department). However, as Sanchez-Graells illustrates, agency theory assumes the opposite: that a procurement arm of government acts as the agent of public buyers such as Welfare Departments, rather than as a constraint on them. A role reversal where the public buyer (the procurement arm) must act as gatekeeper of the public user (the Department) rather than its agent leads to internal governance challenges that procurement law is not equipped to resolve. If, for instance, the procurement arm is institutionally embedded within the organisation that will use the AI, it is unrealistic to think that the principal-agent relationship will be reversed to enable oversight. Furthermore, the ‘decentred interactions’ between the public sector AI user (the Department) and the tech provider may mean that they can jointly shape the effective deployment of AI systems in a way that escapes the influence of procurement law. This may be because of timing (procurement law primarily bites prior to the entry into force of contracts) and the tools available to public procurers (in terms of their technical expertise, for example).

The second assumption that the article challenges is that, where there is AI regulation-by-contract, the public sector acts as the rule-maker with the tech provider as a rule-taker. Sanchez-Graells emphasises how in the absence of detailed public guidance on how to implement AI principles the public buyer is funnelled towards private standards. Dependence on such private standards to substantiate fundamental rights has been criticised in the context of the EU AI Act. The risk is one of regulatory tunnelling, where decision-making power is displaced from the public buyer to the tech provider. The tech provider has the capacity to translate the contract’s requirements into technical and organisational measures based on industry standards (where they exist) or its own preferences (where they do not). Sanchez-Graells also points to the risk of industry shaping standards for commercial gain where regulatory goals are difficult to define or incommensurable.

Following this bleak assessment of the potential for procurement to shape the public use of AI systems, Sanchez-Graells makes the case for institutional reform and the creation of an independent regulator for public sector AI use. This regulator would prevent the public sector from deploying technological solutions that breach fundamental rights and digital regulation principles and would also be tasked with avoiding regulatory capture and commercial determination. To achieve these aims, it would require independence and digital capability. The new regulator would also set mandatory requirements for public sector digitalisation through standard certification and deployment authorisation.

Irrespective of the political feasibility of this institutional reform, through the preceding analysis Sanchez Graells leaves the reader in no doubt that lawyers concerned with public values and private power should be attentive to procurement law. Procurement law may well be the next legal framework to legitimise the expansion and entrenchment of private power in the digital environment, albeit this time at the direct expense of public power.