Lenin did not actually say, “There are decades when nothing happens, and there are weeks when decades happen,” but if he had, he might have been talking about generative AI. Since November 30, 2022 when OpenAI released ChatGPT, decades have happened every week. It’s not just that generative AI models are now able to emit fluent text on almost any topic imaginable. It’s also that every day now brings news of new models, new uses, and new abuses. Legal scholars are scrambling to keep up, and to explain whether and how these AIs might infringe copyright, violate privacy, commit defamation and fraud, transform the legal profession, or overwhelm the legal system.
Samuel R. Bowman’s preprint Eight Things to Know About Large Language Models is an ideal field guide for the scholar looking to understand the remarkable capabilities and shocking limitations of ChatGPT and other large language models (LLMs). Bowman is a professor of linguistics, data science, and computer science at NYU, and a visiting researcher at the AI startup Anthropic. Eight Things is clear, information-dense, and filled with helpful citations to the recent research literature. It is technically grounded, but not technically focused. And if you are paying attention, it will grab you by the lapels and shake vigorously.
LLMs are syntactic engines (or stochastic parrots). What they do, and all they do, is predict the statistical properties of written text: which words are likely to follow which other words. And yet it turns out that statistical prediction — combined with enough data in a large enough model wired up in the right way — is enough to emulate human creativity, reasoning, and expression with uncanny fluency. LLMs can write memos, program games, diagnose diseases, and compose sonnets. Eight Things is a thoughtful survey of what LLMs can do well, what they can’t, and what they can pretend to.
Bowman’s first two Things to Know are an unsettling matched pair. On the one hand, LLMs predictably get more capable with increasing investment, even without targeted innovation. Simply pouring more time, training data, and computing power into training an LLM seems to work. This means that progress in the field is predictable; at least for now, it doesn’t seem to depend on uncertain scientific breakthroughs. Decades will keep on happening every week. (Indeed, the sixth Thing to Know, human performance on a task isn’t an upper bound on LLM performance, means that there is no necessary limit to this progress. For all we know, it might be able to continue indefinitely.)
But on the other hand, specific important behaviors in LLM[s] tend to emerge unpredictably as a byproduct of increasing investment. The fact of progress (Gozer the Gozerian) is predictable, but not its specific form (the Stay Puft Marshmallow Man). As Bowman explains, part of what makes ChatGPT so powerful and so adaptable is that it displays few-shot learning: “the ability to learn a new task from a handful of examples in a single interaction.” Post-ChatGPT LLMs are not just purpose-built AIs with fixed capacities — they can be coached by users into competence on new tasks. This is why, for example, ChatGPT can produce baseline-competent answers to law-school exams, even though almost no one had “go to law school” on their bingo cards five years ago.
Bowman’s third Thing, LLMs often appear to learn and use representations of the outside world, is also remarkable. Even though they are only syntactic engines, LLMs can give instructions for drawing pictures, draw inferences about the beliefs of a document’s author, and make valid moves in board games, all tasks that are usually thought of as requiring abstract reasoning about a model of the world. Legal doctrine and legal theory will need to decide when to adopt an external perspective (“ChatGPT is an algorithm for generating text, like throwing darts at a dictionary”) and when to adopt an internal one (“ChatGPT acted with actual malice when it asserted that I committed arson”).
Unfortunately, there are no reliable techniques for steering the behavior of LLMs. While Bowman describes widely-used techniques for steering LLM behavior—crafting well-chosen prompts, training on well-chosen examples, and giving human feedback—none of these techniques are reliable in the way that we typically think a well-trained human can be. While LLMs are getting better at learning what humans want, this is not the same as doing what humans want. “This can surface in the form of … sycophancy, where a model answers subjective questions in a way that flatters their user’s stated beliefs, and sandbagging, where models are more likely to endorse common misconceptions when their user appears to be less educated.”
The seventh Thing—LLMs need not express the values of their creators nor the values encoded in web text—expands on this depressing framing to explore specific ways in which researchers are trying to embed important legal and societal values in LLM outputs. As programmer Simon Willison has argued, it is hard or impossible to put guardrails around an LLM to prevent it from producing specific kinds of outputs. Malicious users with sufficient dedication and creativity can often use “prompt injection” to override the developer’s instructions to the LLM system with their own.
One reason that steering is so difficult is due to Bowman’s fifth Thing: experts are not yet able to interpret the inner workings of LLMs. Legal scholars have been writing thoughtfully about the interpretability problem for AIs. Giving high-stakes decisions over to an AI model offends important rule-of-law values when the AI’s decisions cannot be more intelligibly explained than “computer says no.” LLMs and other generative AIs compound these problems. The legal system currently depends on an ability to make causal attributions: was a fraudulent statement made with scienter, or was the defendant’s work subconsciously copied from the plaintiff’s? The current state of the art in AI research gives us very little purchase on these questions.
Bowman’s eighth and final point is a further reinforcement of the limits of our current knowledge: brief interactions with LLMs are often misleading. On the one hand, the fact that an LLM currently trips over its own feet trying to answer a math problem doesn’t mean that it’s incapable of answering the problem. Maybe all it needs is to be prompted to “think step by step.” LLMs and high-schoolers both benefit from good academic coaching. On the other hand, the fact that an LLM seems able to execute a task with aplomb might not mean that it can do as well on what humans might consider a simple extension of that task. It turns out, for example, that GPT-4 memorized coding competition questions in its training set: it could “solve” coding questions that had been posted online before its training cutoff date, but not questions posted even just a week later.
LLMs are strikingly powerful, highly unpredictable, prone to surprising failures, hard to control, and changing all the time. The world urgently needs the insights that legal scholars can bring to bear, which means that legal scholars urgently need to understand how LLMs work, and how they go wrong. Samuel Bowman’s insightful essay is table stakes for participating in these debates.
