The most common way to evaluate LLMs is to pit them against a human baseline, or against each other, on benchmarks – standardized tests or sets of tasks designed to assess specific capabilities of these models. Benchmarks are designed to provide a quantitative assessment of model performance in various domains, facilitating a direct comparison of their abilities in a controlled and systematic way. Unfortunately, benchmarking methods are plagued by limitations that hamper their reliability and adequacy to arbitrate dispute about the capacities of LLMs. These limitations include saturation, gamification, contamination, and lack of construct validity (which, as we shall see, are not all independent concerns, but are interrelated).

New natural language processing (NLP) benchmarks tend to saturate at an accelerating pace, meaning that LLMs quickly surpass the human baseline [33, 34]. This is particularly striking for so-called ‘natural language understanding’ (NLU) benchmarks. Taken at face value, saturation on these benchmarks would suggest that the best-performing NLP models ‘understand’ language better than humans themselves. However, such success is often marred by independent examples of obvious failures, suggesting that performance saturation is not reliable evidence that LLMs actually surpass humans on the cognitive ability or aptitude (the ‘target construct’) that the benchmark was designed to assess. In the case of NLU, the target construct – language ‘understanding’ – is itself ill-defined. See our discussion of semantic competence in Part I [24] and our discussion of construct validity below.

Several factors explain benchmark saturation. One such factor is the gamification of benchmarks. Intense competition for best model performance on leaderboards – also known as ‘SOTA’ (state-of-the-art) chasing – ultimately undermines the very purpose of behavioral evaluation. It results in models optimized to improve scores on proxy metrics targeted at benchmarks rather than the attribute the benchmark was designed to assess. This can be seen as an example of Goodhart’s Law, commonly stated as the idea that when a measure becomes a target, it thereby ceases to be a good measure [35]. In the context of benchmark creation, statistical relationships identified from empirical observation and used to construct proxy metrics measured by test items tend to break down when actively exploited and optimized [36]. This results in models that may overperform on a benchmark, yet lack the capacity supposed to be measured by that benchmark, at least to the degree suggested by their performance. See [1] for an example of how artifacts in benchmark construction can amplify the divergence between proxy metrics and target construct as models saturate the benchmark.

Another phenomenon that contributes to benchmark saturation is data contamination. This occurs when test data – including benchmark items and their solutions – leak into the training data, either by chance or by design. With LLMs trained on internet-scaled data, contamination is increasingly difficult to avoid because benchmarks are widely reproduced and discussed online [32]. Detecting contamination is a challenge because simple variations of test items, such as paraphrases, can easily evade common detection measures based on string matching [37]. This calls into question the significance of some LLMs’ reported performance on certain tests. For example, there is evidence that GPT-4’s training data was contaminated with solutions to problems from the competitive programming contest Codeforces, such that it performs much worse than reported on old or recent problems that did not make it into its training data [38].

A broader concern with LLM benchmarks pertains to the construct validity of their target. Construct validity refers to the degree to which a test accurately measures the theoretical construct it purports to assess. Standardized tests designed for humans are (ideally) meaningful because they are conceived to measure some underlying skill or capacity, such that good test performance should generalize to relevant real-world situations. However, establishing construct validity for LLM benchmarks is more challenging. Constructs targeted by common benchmarks, such as ‘understanding’ or ‘reasoning,’ are often abstract, multifaceted, and implicitly defined with reference to human psychology. Simply using test items validated for human subjects will not do, not only because of the potential for contamination, but also because validity is conditional upon background theoretical assumptions about test subjects. In other words, background assumptions about human cognition on which human-centric tests are premised can impact their applicability to LLMs. As a result, an LLM achieving good performance on a validated test designed to measure a capacity \( \phi\) in humans may not constitute adequate evidence that the LLM has \( \phi\) .

As an example, consider the contentious case of Theory of Mind (ToM) – the cognitive ability to understand and attribute mental states to oneself and others, and to recognize that these mental states may differ from one’s own. Using classic verbal false belief tasks adapted from the developmental psychology literature on ToM, [2] found that GPT-3 performed comparably to nine-year-old children. However, [3] complicates this picture by showing that introducing minor conceptual variations on these test items, while preserving the requirement for false belief inference, significantly degrades LLM performance. This disparity suggests that GPT-3’s impressive performance on original test items does not provide adequate evidence of ToM. This could certainly be due to some form of data contamination – after all, there is hardly any doubt that false belief tasks feature in GPT-3’s training data. But [3] also suggests that even if LLMs matched human performance on challenging variations of classic false belief tasks, we should be careful not to jump to conclusions about their putative ToM aptitude. Indeed, tests designed with human cognitive processes in mind warrant inferences about the target construct given plausible assumptions about the mechanisms that drive human performance on these tests. These assumptions may not translate to artificial systems like LLMs without independent motivation. For example, the developmental trajectory and cognitive architecture of human ToM may lend itself to simulation-based or causal models of other minds that generalize flexibly. In contrast, an LLM could potentially achieve similar task performance by memorizing and interpolating patterns in training data, without engaging in the same underlying reasoning. Thus, the construct validity of ToM tests depends not just on input-output mappings, but on the nature of the algorithms and representations that generate those outputs. We have independent reasons to posit Theory of Mind as a core socio-cognitive capacity in humans, based on its early emergence, specificity to agents, and dissociability from general intelligence. Analogous supplementary evidence might be needed to corroborate claims about emergent ToM in LLMs.

Most of the limitations of LLM benchmarking are inherent to behavioral evaluation more generally. This point is often highlighted with reference to the distinction between performance and competence [39, 40]. The distinction is most commonly invoked by skeptics about LLMs to suggest that performance success, such as high scores of benchmarks, need not reflect the competence associated with the underlying construct [33]. Humans may achieve good performance because they exercise some cognitive capacity \( \phi\) (e.g., ToM), while LLMs may achieve similar performance through completely different means (e.g., memorization of linguistic patterns that correlate with common ToM evaluations). The core concern here is that input-output mappings provide insufficient evidence about the mechanisms that lead from input to output in a given system. Consequently, serious research on the capacities of LLMs should go beyond behavioral evaluation, and seek to understand how they process information mechanistically.

It should be noted that the performance-competence distinction cuts both ways. In humans, it is common to disregard performance failures as evidence of a lack of competence, because such failures may be explained away by contingent constraints on cognitive function (e.g., limitations of working memory or attention). Most famously, Chomskyan linguists argue that performance errors are mostly irrelevant to linguistic competence, because they are the result of processing limitations or external factors rather than a reflection of the underlying linguistic system. There is much debate about how to delineate the performance/competence distinction, as well as some confusion which is partly due to somewhat inconsistent characterizations in Chomsky’s work itself. For an enlightening exposition of the distinction in linguistics, see [4]. For a discussion of its relevance to the NLP and LLMs, see [5] and [6]. Many cognitive scientists have called into question the existence of such an absolute gap between performance and competence [41, 42]. Another concern is that hypotheses about performance limitations need independent empirical support (lest they be invoked ad hoc to preserve a classical model in the face of empirical disconfirmation), but this independent evidence is rarely provided even in the case of humans, and direct investigations of performance factors can clash with competence theories [43, 41, 44]. Nonetheless, it is intriguing to consider how the dissociation between performance and competence might also apply to LLMs in both directions. That is, just as LLMs’ impressive performance on benchmarks may always not accurately reflect their true competence, one might also consider that the existence of some performance errors do not always conclusively establish the absence of such competence.

In science, mechanistic explanations aim to reveal the causal structure underlying a phenomenon of interest by describing the organized entities and activities that are responsible for producing or maintaining that phenomenon [45]. More precisely, a mechanistic explanation identifies the component parts of a mechanism, characterized by their properties and capacities, the causal interactions between these component parts, and the organization of the parts and activities such that they give rise to the phenomenon. Such explanations stand in contrast to purely descriptive or phenomenological models that simply re-describe the phenomenon itself, as well as covering-law explanations that explain by subsuming the phenomenon under empirically discovered regularities or governing laws. Mechanisms explain by revealing how the phenomenon arises from the causal structure of the system, not merely describing empirical regularities in which it partakes. As such, mechanistic explanations reveal opportunities for manipulation and control over the phenomenon in a way that descriptive or law-based explanations do not.

Understanding the behavior of a simple system like a mechanical clock is easy enough – one can simply open it up and observe the mechanism at work. This is not so straightforward with more complex systems, like the weather, the brain, or artificial neural networks. Neural networks are often described as ‘black boxes’ precisely because the causal mechanisms that explain their behavior seem opaque to scrutiny. As we emphasized in Part I [24], simply staring at the learning objective, architecture, or parameters of LLMs will reveal neither how they exhibit their remarkable performance on challenging tasks, nor what functional capacities can be meaningfully ascribed to them. In principle, one could even provide a complete mathematical description of an LLM as a giant composite function, consisting in an absurdly complex sequence of linear and nonlinear transformations across many layers; but such a description, on its own, would be useless to provide a genuine explanation of the network’s behavior in specific contexts. The ‘black box’ metaphor underscores this chasm: it highlights the difficulty to trace precise causal pathways in the network through which specific inputs are transformed into specific outputs. This is why merely re-describing what an LLM does in terms of next-token prediction or matrix multiplication – what we called the ‘Re-description Fallacy’ in Part I – cannot possibly settle philosophical debates that are fundamentally about causal mechanisms.

The search for causal mechanisms has become central across the life sciences and cognitive science. Like their artificial counterparts, biological neural networks are ‘black boxes’; yet neuroscientists are engaged in the project of uncovering multilevel mechanisms underlying psychological capacities and nervous system functions [46]. At the molecular and cellular levels, they describe mechanisms of protein synthesis, gene expression, and synaptic transmission; at an intermediate level are mechanisms of neuron spiking and oscillation as well as mechanisms of development and synaptic plasticity; at higher levels are mechanisms underlying learning, memory, reasoning, and other psychological capacities.

Mechanisms explain by opening up the causal ‘black box’ linking cause and effect – revealing the internal entities, activities and organization that transmit causal influence through the system. The notion of intervention is central to this explanatory project. In the philosophy of science, interventionism holds that causal relationships are best understood in terms of what would change under interventions or manipulations to parts of the system [47]. More specifically, \( X\) is considered a direct cause of \( Y\) if and only if there exists a possible intervention on \( X\) that will change \( Y\) (or the probability distribution of \( Y\) ) when all other variables in the system are held fixed. This relationship is asymmetric – intervening on \( Y\) does not change \( X\) if the causal arrow truly points from \( X\) to \( Y\) . Interventionism eschews regularity or correlational notions of causation, recognizing that a system’s behavior depends on more than merely observing regular successions of events. In the context of mechanistic explanation, interventions involve altering specific parts of a posited mechanism piecemeal in order to learn about their causal contribution to the target phenomenon. For example, pharmacology targets specific receptor mechanisms or cell signaling pathways; brain stimulation techniques activate or inhibit activity in restricted brain areas; optogenetics uses light to control neurons genetically modified to express light-sensitive channels; and knockout models remove genes hypothesized to be critical for mechanisms.

A similar interventionist approach can be applied to unravel causal mechanisms in artificial neural networks like LLMs. Of course, there are major disanalogies between biological nervous systems evolved over millions of years and artificial neural networks designed by human engineers. Nevertheless, the motivation of interventionist research is similar: to achieve explanatory understanding by revealing multilevel mechanisms, not merely observing input-output patterns. Like neuroscientists, computer scientists can aim for explanatory understanding linking particular components of neural networks and patterns of internal activations to specific capacities, such as translating between languages or answering arithmetic questions. This explanatory project is often described as mechanistic interpretability. In a broad sense, mechanistic interpretability could be characterized as the search for mechanistic explanations of the behavior of deep neural networks, including LLMs. As we shall see, however, the phrase is often used in a more restrictive sense, to denote a particular set of theoretical assumptions and intervention methods used to achieve mechanistic explanations in machine learning. The lack of a common lexicon in interventionist research on neural networks is rather unfortunate – many different labels exist for similar ideas and methods, with ‘interpretability’ being a particularly confusing term [48]. In what follows, we will attempt to give a cohesive overview of this fragmented literature.

There are three main methodological approaches to investigate the inner structure of neural networks: probing, attribution, and bona fide causal intervention. Probing involves training a separate supervised classifier, known as a diagnostic probe, to predict certain properties (e.g., part-of-speech tags, dependency relations) from the model’s internal activations [49, 50]. High accuracy in decoding a particular linguistic feature \( F\) from a probe tuned to a pattern of activation \( A\) in a subset of the network can be though to provide evidence that \( A\) is sensitive to \( F\) , and provide information about its presence or absence for downstream processing. However, a probe’s successful prediction of a certain linguistic feature from a model’s activations does not necessarily mean that the feature plays a causal role in the model’s behavior [51]. In addition, probes can pick up on spurious correlations, thereby failing to distinguish between genuine representation and incidental associations [52].

Probing can tell us that some information is likely present in the activations of the system, but not that it is in fact used for a particular purpose or function in generating the system’s outputs. Indeed, the mere presence of usable information does not demonstrate that it is actually exploited. By way of analogy, suppose you throw block letters spelling a word into a pond. With the right apparatus, you might be able to decode the word’s identity from the ripples formed by letters over the pond’s surface; but this does not provide evidence that the pond represents (let alone ‘understands’) the word in any meaningful sense. It merely shows that the block letters created surface patterns whose origin could be recovered by a sufficiently powerful decoding method.

Methods that do not involve training a separate classifier – also known as nonparametric methods – have been explored as an alternative to probing. For example, one can directly investigate the weights (or attention scores) that attention heads place on different parts of the input sequence. A high attention score on a particular word might be thought to provide evidence that the head is playing a role in processing the semantic or syntactic role of that word in the rest of the sentence. This approach to interpretability falls within the broader and somewhat loose umbrella of attribution methods in deep learning. Attribution methods assign importance scores to input features to explain individual model predictions; in other words, they are meant to identify parts of the input that are most influential for the output. While attention patterns in Transformers intuitively provide clues about the relevance of each input token to model predictions, simply analyzing these patterns in a given layer of the network only offers limited explanatory power [53]. In particular, visualizing attention patterns is thoroughly insufficient to draw strong conclusions the flow of information through the network. If we want to understand what information the model represents in the input, and how that information is processed through step-by-step computations across layers to drive output predictions, we need to intervene directly on the network to reveal its causal structure.

To assert that a certain activation pattern in a model genuinely represents a given feature, mainstream philosophical theories of representation require that three criteria be satisfied: (a) the activation pattern should carry information about the feature; (b) the activation pattern should influence the model’s behavior in a task-relevant way, and (c) the model should be capable of misrepresenting the feature [54]. Standard probing methods only provide evidence for the first criterion. Showing that some information about a feature is actually used by the LLM to generate its outputs requires suitable interventions on patterns of activation encoding that information. Changes in model behavior caused by such interventions should be consistent with the hypothesis that the model represents the target feature, but should not be explainable by mere perturbations from the training set. Accordingly, an increasingly large body of work uses targeted intervention methods to establish causal relationships between language models’ internal representations and their behavior.

The simplest kind of intervention on neural networks – both artificial and biological – is an ablation. In the context of artificial neural networks, ablation involves disabling or eliminating individual neurons or groups of neurons within a trained model to observe the resulting changes in behavior. By ablating neurons and observing the resulting change in performance metrics such as classification accuracy or reconstruction error, researchers can determine how much each neuron or group of neurons contributes to the network’s overall function. Neurons that are critical to network performance will result in a substantial decline in metrics when ablated.

Indiscriminate ablations, common in the early days of connectionist research, involved disabling nodes at random. This approach was aimed at demonstrating general properties of neural networks. One key finding from such studies was the concept of graceful degradation: like biological brains, the performance of neural networks tends to decline in a gradual, rather than abrupt, manner when parts of the network are damaged or disabled [55, 56].

Targeted ablations, on the other hand, involve disabling specific nodes or modules believed to serve distinct representational or functional roles within the network. By analyzing the network’s altered behavior, researchers could infer the impact of the disabled nodes, thereby gaining insights into their hypothesized functions [57]. This method mirrors a common approach in neuroscience where the study of natural or induced brain lesions helps to unravel the functions of specific brain areas.

However, both indiscriminate and targeted ablation studies are somewhat limited in their ability to fully uncover representational or functional roles in neural networks. This limitation is partly due to the nature of how information is represented within these networks. Traditional views centered around localized representations, positing that specific neurons or small groups of neurons could be responsible for representing distinct, complex stimuli or concepts. The classic example in neuroscience is the hypothetical ‘grandmother cell’ – a neuron that would activate exclusively in response to the mental image or concept of one’s grandmother. While intuitively appealing, this model of localized representation has largely been abandoned in favor of a distributed model [58, 59].

In practice, individual neurons or small clusters of neurons in neural networks often encode information about multiple concepts [60, 61, 62]. This is consistent with distributed representations: concepts are represented by patterns of activity that are spread across a larger number of neurons within the network. When components of distributed representations overlap, the same neurons might simultaneously represent multiple features – they are ‘polysemantic’. This property is studied under the name of ‘superposition’; for example, [7] demonstrate that superposition can allow networks to linearly represent many more features than they have neurons or dimensions, at the cost of some interference between features. The distributed model of representations in neural networks reflects a more integrated and holistic approach to neural processing, where information is not stored in isolation but as part of a dynamic, interconnected system.

The shift from localized to distributed representation models has significant implications for interpreting the results of ablation studies. In a distributed system, disabling a node or a set of nodes affects more than just the immediate functionalities those nodes are associated with. It impacts the network-wide interplay of neural activities. Therefore, determining the specific effects of localized damage on the overall behavior of the network becomes increasingly complex [63].

Fortunately, the level of control we have over artificial neural networks allows for more sophisticated causal interventions that improve on localized ablations by allowing us to ‘edit’ distributed representations in ways consistent with representational or functional hypotheses, and verifying corresponding changes in behavior. For example, [8] used a probe to identify activations associated with subject-verb number agreement, then modified these activations to improve the model’s performance on a subject-verb agreement task. Such interventions provide more robust evidence about the causal role of specific internal features in a model’s behavior.

A more sophisticated approach called ‘iterative nullspace projection’ was developed by [9]. Iterative nullspace projection can determine whether some particular information is causally involved in a neural network’s predictions by identifying and removing that information from distributed neural representations, and then assessing the consequence on model behavior. The approach involves iteratively projecting neural representations onto the nullspaces of a probe to remove detectable information about user-defined target concepts.

More specifically, the first step is to train linear probe on internal neural representations to predict values of the concept of interest. For example, probes could be trained to classify grammatical number from the hidden state of a language model. The nullspace of a probe is the subspace consisting of all activations for which the probe makes the same constant prediction. Projecting representations onto these nullspaces removes information that linearly correlates with concept values while preserving unrelated information. After projecting onto the first probe’s nullspace, a second probe is trained on the transformed representations to predict the target concept. This process is repeated iteratively, with projections onto each new probe’s nullspace removing additional detectable information about the target concept. Finally, the network makes predictions using the fully projected representations where linear information about the target concept has putatively been eliminated. If performance degrades, the target concept can be inferred to be causally significant for model performance, suggesting that the network exploits it for its original computations.

For example, [10] investigated whether information about relative clause boundaries is used by language models like BERT to predict subject-verb agreement across a relative clause. They trained a set of linear probes to predict whether a word is inside a relative clause, based on the model’s contextual word embeddings. Each probe defines a direction in the representation space that separates words inside relative clauses from words outside relative clauses. Collectively, these directions span a ‘feature subspace’ that contains relative clause information. The orthogonal complement of this subspace is the nullspace which does not contain information useful for predicting whether a word is inside a relative clause. Then, iterative nullspace projection is used to generate ‘counterfactual representations’ for the masked verb token. The representation is projected into the relative clause feature subspace, and then flipped to the opposite side of the separating hyperplane, either towards the side containing relative words (‘positive counterfactual’) or away from that side (‘negative counterfactual’). This process minimally modifies the representation to incorrectly encode that the word is inside or outside a relative clause. Finally, the effect of the counterfactual representations on the model’s number agreement predictions is measured. If swapping in the positive counterfactual increases error rate and the negative counterfactual decreases error rate, this alignment with predictions from linguistic theory suggests the model uses relative clause boundary information appropriately for agreement, confirming the causal effect (fig. 1).

Pan/Zoom

This method is directly inspired by counterfactual approaches to causal explanation in philosophy of science [47]. Such approaches aim to isolate the causal contribution of some factor \( X\) to an outcome \( Y\) by minimally altering the value or presence of information about \( X\) in the system, while holding all other factors fixed. This allows assessment of whether and how much the factor \( X\) is exploited causally by the system to produce \( Y\) . The removal of detectable information about a target variable through iterative nullspace projection is analogous to a hypothetical intervention that breaks the links between that variable and the system while leaving other causal mechanisms intact. The comparison between original model outputs and outputs based on projected representations missing information about \( X\) mirrors the evaluation of interventionist counterfactuals, which aims to answer questions such as “What would happen to \( Y\) if an intervention prevented information about \( X\) from influencing the mechanism?”.

Intervention methods such as iterative nullspace projection can help us determine whether some target concept, such as a particular syntactic feature, is represented by a language model and causally efficacious in its behavior. But this is insufficient to understand language models and other neural networks the way we understand classical computer programs. Indeed, the guiding ideal of interpretability research is to model the internal causal structure of neural networks, such that we can explain some behavior of interest in terms of a series of computational steps applied on the input. Mechanistic interpretability refers to this concerted effort to reverse engineer the internal computations performed by artificial neural networks. Rather than solely focusing on interpreting individual model predictions, it aims to provide a detailed and systematic understanding of how the model transforms inputs to outputs [64]. The overarching goal of mechanistic interpretability is to open up the ‘black box’ of neural networks by providing human-intelligible descriptions of the functional modules that drive the emergence of model behaviors.

Like systems neuroscience, mechanistic interpretability specifically targets the algorithmic level of analysis [65, 66]. An algorithmic explanation elucidates the flow of information through a network and the sequence of operations performed upon it. Crucially, it abstracts away from engineering (or biological) details, instead focusing on the computations performed by the system. Algorithmic explanations support counterfactual inferences about how changing inputs or network components would affect computations and outputs. This involves identifying structural elements like motifs and circuits that are reusable, such that manipulating them produces systematic effects across many inputs and conditions.

Mechanistic interpretability specifically seeks to reverse-engineer neural networks in terms of learned features and reusable circuits that operate on those features. Features refer to human-interpretable properties of the input data that the model represents internally. For computer vision models, such features might be edges, textures, or shapes. For language models, features may correspond to part-of-speech tags, named entities, or semantic relationships. By studying a network’s internal activations, researchers aim to determine which features are encoded where.

Circuits, in turn, are chains of operations that detect certain input features and transform them into output features. The goal is to decompose an entire neural network into a hierarchy of understandable features and circuits that process information step-by-step. This modular view would explain how trained models transform inputs to outputs via learned features and program-like circuits, providing a form of algorithmic understanding. In fact, it is even possible to program a Transformer from scratch using a specialized programming language like RASP (Restricted Access Sequence Processing Language). RASP allows mapping the basic components of a Transformer, like attention and feed-forward layers, into simple primitives that can be composed into programs. These RASP programs can then be compiled into the weights of a Transformer network that implements the specified computation [67]. More recently, methods have been developed to train Transformers whose weights are constrained to implement human-interpretable RASP-like programs. These Transformer Programs can be learned end-to-end from data and decompiled back into discrete, readable code [68]. Such approaches provide a more direct way to understand the computations of a Transformer in terms of modular algorithmic components.

More formally, a given neural network can be modeled as a graph (sometimes called a computational graph) whose nodes correspond to components of the network at some level of granularity – such as attention heads or individual neurons. The directed edges between nodes represent the flow of information and computations in the neural network. The connectivity defined in the computational graph needs to faithfully represent the actual computation flow in the neural network. In this model, a circuit is a subgraph within the overall computational graph that implements some specific behavior or functionality of interest. The goal of representing the neural network as a computational graph is to systematically analyze it and localize particular circuits that are causally responsible for certain model behaviors.

Once candidate circuits are found, their functionality must be validated through causal interventions. This often involves surgically replacing components of suspected circuits and observing the effect on model outputs. Validated circuits can then be composed into a hierarchical understanding of the succession of transformations enacted by the model. The resulting mechanistic explanation should provide significant behavioral control, for instance allowing targeted editing of model computations.

The picture that emerges from mechanistic interpretability research is that Transformer models can be viewed as containing parallel processing streams (also known as ‘residual streams’), one at each input token position (fig. 2). While each residual stream encodes information in a very high-dimensional vector space, attention heads at each layer operate over much smaller (low-dimensional) subspaces of the stream that may not overlap with one another. Making use of these disjoint subspaces allows Transformers to route information about tokens and their dependencies dynamically across layers and positions. Specifically, each stream is functionally analogous to a addressable memory, in which attention heads can write to and read from subspaces of the main embedding space. Such information may include, for example, syntactic dependencies between tokens in the input sequence.

Pan/Zoom

A common intervention method used in mechanistic interpretability research is activation patching [69], also known as causal tracing [70] and interchange intervention [71] (fig. 3). The basic method involves three steps. First, the neural network must be run on an original input that relates to some behavior of interest (e.g., answering factual questions). For example, the original input might be “The capital of France is...”, with the expected output being “Paris”. Importantly, the activations of the network during the forward pass on this original input are cached for later use. The second step is to run the model again with an alternative input that introduces a key variation on the original input that changes the behavior (output). For example, an alternative input might be “The capital of Germany is...”, with the expected output being “Berlin”. Finally, the alternative input is run again but a specific component of the network’s activation is swapped for its cached value from the original forward pass – an intervention known as ‘patching’, because it patches the alternative forward pass with a component of the original forward pass. The effect of patching a component’s activation is then evaluated by comparing the model’s performance in the regular forward pass on the alternative input versus the patched forward pass on the same input. For example, the metrics used could be the probability the model assigns to the original, correct token (“Paris”), or the difference in logits between the correct and incorrect tokens (“Paris” vs “Berlin”). Intuitively, changing the input hurts model performance on the expected behavior, while patching activations from the original run helps restore it. So if patching a particular component leads to a significant restoration of performance (e.g., an increase in the probability of “Paris” being the output), it suggests that component is important for the model’s behavior on the task (i.e. the component not only contains information on the target feature, but that information is causally implicated in producing the desired result). By iterating this procedure over many components of the computational graph, such as attention heads in a Transformer model, activation patching aims to identify the key circuits that enable behaviors like factual recall or logical reasoning.

Pan/Zoom

The mechanistic interpretability framework has been applied to many different problems, and has started shedding light on some abilities of Transformers. It is worth noting that much of this research is painstaking work conducted with toy models that are easier to interpret. While these toy models are based on the Transformer architecture, they may differ from LLMs in terms of architectural details, learning objective, dataset, and of course size (parameter count). Nonetheless, efforts are underway to automate and scale mechanistic interpretability techniques to bona fide LLMs, with promising initial results [72, 73, 74]. In what follows, we will briefly illustrate the fruitfulness of the mechanistic interpretability program through three case studies.

2.5.1 Case study 1: Induction heads

A canonical example of circuit discovery in language models through the tools of mechanistic interpretability is that of so-called ‘induction heads’ [11]. These are specialized attention heads that emerge through training even in very small models, and implement a form of pattern completion, allowing models to repeat or generalize sequences based on similar context patterns.

Specifically, induction heads use a ‘prefix matching’ attention pattern to look back over previous tokens and detect if any match the current token. Rather than relying merely on memorized statistics about which tokens tend to follow others, they flexibly attend to whichever prior token is most similar to the current one based on learned representations. If a previous token is sufficiently similar, the induction head will attend to the next token after it. The head then increases the probability of that attended next token, effectively predicting that the current sequence will continue like the previous matched sequence. For example, if the input contains the sequence “...the cat sat on the mat. The cat...,” the induction head matches the second “cat” token to the first one, attends to ”sat” from the first sequence, and increases the likelihood of outputting “sat” again. This allows Transformer models to repeat sequences and generalize patterns.

Importantly, the computations performed by induction head circuits do not rely on bigram statistics memorized from the training data; rather, they operate over abstract patterns in the input sequence (prompt), even if the latter does not contain familiar strings. As such, induction head circuitry can be seen as an instance of what [12] called non-content-specific computations in neural networks: computations that apply the same procedure or algorithm irrespective of the specific content represented at input and output. Specifically, when the network is processing a sequence [A][B]...[A], the induction head circuit can be characterized algorithmically as storing the value of the first [A] token in a specific subspace of the residual stream – which we may call the previous_token subspace – at the position of the [B] token. Importantly, this operation occurs whatever that specific value (content) of [A] might be. Functionally, the previous_token subspace in which this information is stored operates rather like a variable in a classical symbolic program. Its value is accessed at the next layer by the second part of the induction head circuitry, which reads the content of the previous_token subspace in the residual stream at the position of the [B] (fig. 4). This is somewhat analogous to the indirect addressing mechanism that enables variable binding in classical systems: an attention head stores information about the token that precedes another token in a dedicated subspace of the residual stream (what we called the previous_token subspace), such that a distinct attention head in a later layer may retrieve it for downstream processing. This kind of circuit satisfies the distinction between storage and use that is crucial to computation over bound variables [56, 75]. For a discussion of induction heads circuitry as implementing a form of variable binding, and the implications of this view for the debate about compositionality in connectionist models, see [13].

Pan/Zoom

Circuits using induction heads may be an important building block for the advanced in-context learning abilities exhibited by LLMs. Even in toy models, for instance, if the context contains “...the cat sat on the mat. The dog...”, an induction head can generalize that “dog” signals repeating the sequence, despite never seeing “dog” next to “sat” during training. Research shows that Transformer models excel at discovering complex abstract patterns in context and extrapolating from them [14].

2.5.2 Case study 2: Modular addition

Mechanistic interpretability is not just helpful to investigate how trained neural networks process information through algorithmic circuits, but also how they learn such algorithms during the course of training. Studying the training dynamics of neural networks is important, because it can provide insights into learning transition phases that are highly relevant to ongoing debates about the capacities of LLMs. Thus, [15] investigated the puzzling phenomenon of grokking, where neural networks trained on algorithmic tasks with regularization initially overfit to the training data but later suddenly generalize after many training steps. As a case study, the authors trained small Transformer models on modular addition tasks. They find these networks exhibit grokking, initially overfitting but later learning to generalize.

To understand this phenomenon, they reverse engineered the mechanisms learned by these networks using techniques from mechanistic interpretability. That found that the network learns to perform modular addition tasks by mapping inputs onto rotations in the plane and composing those rotations using trigonometric identities (fig. 5). This clever algorithm, dubbed ‘Fourier multiplication,’ allows the network to perform addition modulo the prime.

Pan/Zoom

Using this understanding, Nanda et al. defined new progress metrics allowing them to study the training dynamics of the models as they learn to perform modular addition algorithmically. They identified three distinct learning phases, each marked by continuous progress on certain metrics:

1. Memorization phase: In the early part of training, the models fit to the training data by simply memorizing input-output pairs. Performance on the test set remains low while performance on the training set increases rapidly, indicating the models are overfitting.
2. Circuit formation phase: After memorization, there is a transition period where the models internalize the algorithm for modular addition using trigonometric identities and rotations (the ‘Fourier multiplication’ circuit). However, performance on the test set remains low, implying memorization components still persist.
3. Cleanup phase Finally, weight decay drives the removal of the initial memorization components. Performance on the test set abruptly improves to match performance on the training set at the end of this phase, corresponding to the ‘grokking’ transition in generalization capability.

The entire phenomenon is thus not a sudden onset of generalization ability, but rather a gradual amplification of structured mechanisms encoded in the weights, followed by pruning of unnecessary components. Importantly, these findings speak against the skeptical view of LLMs discussed in Part I [24], according to which they are analogous to giant lookup tables with memorized input-output pairs. While the initial training phase does support that view, subsequent phase transitions show that Transformer models are perfectly capable to learn general, rule-like algorithms to solve tasks, including tasks as rigid and well-defined as arithmetic problems.

2.5.3 Case study 3: World models

As we discussed in the complement paper [24], there is an ongoing debate about whether LLMs possess world models. While behavioral evaluations are generally not helpful to settle this debate, mechanistic interpretability shows promise to uncover what the LLMs and related Transformer models represent internally.

To investigate whether Transformer models trained to predict sequences can learn interpretable world representations rather than mere surface statistics, [16] focus on the board game Othello as a simplified yet non-trivial domain. They trained a GPT variant (Othello-GPT) to sequentially predict tokens representing Othello board positions. The only inputs to the model are move sequences derived from game transcript; no explicit game rules or board structure are provided. After training on championship games and synthetic games, Othello-GPT recommends legal moves with high accuracy, suggesting it has learned more than surface statistics. Furthermore, nonlinear probes reliably predict board states from model activations, implying that a nonlinear representation of the board state had emerged during training. To validate the probes’ accuracy, the authors performed intervention experiments that modify Othello-GPT’s internal activations to reflect altered board states. The model’s subsequent move predictions change accordingly, confirming the causal role of these latent board state representations.

In a follow-up study, [17] discovered that rather than representing the board state (e.g., representing each board tile as black, white, or empty), Othello-GPT actually encodes tiles relative to the current player (as player, opponent, or empty). By re-orienting probes to classify this player-centric representation, Nanda et al. demonstrated that the board state is in fact linearly encoded with high accuracy in the network, contrary to [16]’s claim that the board state is only encoded non-linearly (fig. 6). They further demonstrated behavioral control by conducting simple vector arithmetic interventions to alter the model’s encoding of board states and change predictions accordingly. [18] found similar evidence that information about board state is encoded in a simple, linear way in the deeper layers of Othello-GPT models. Like [17], they decoded a representation corresponding to tiles marked player, opponent, or empty, which aligns well with the role of the model in alternating between playing as white or black. To test whether these internal representations play a causal role in the model’s predictions, they also intervened by manipulating activations to trick the model about the state of the board. Through visualizing effects on predicted logits and comparing distributional similarity of logit outputs, they demonstrated layers in which this internal representation steers next-move predictions. The internal representation appears fully developed and utilized in middle layers of deeper models, while shallow models fail to use the representation causally.

Pan/Zoom

These findings align with the linear representation hypothesis, according to which high-level concepts or features are represented linearly as directions in a neural network’s activation space. This hypothesis is often taken to be central to the agenda of mechanistic interpretability [7]. Indeed, if features correspond to directions, we can in principle extract and understand them by projecting activations onto those directions. This should allow us to decompose the high-dimensional activation space of the network into interpretable components. We can also check whether the weights of downstream circuits align with a feature vector and thus conclude that the model is using this feature. Linear directions provide a global foothold across the whole model to reason about a specific feature. Furthermore, manipulating model behavior via causal intervention becomes significantly simpler when features are linear directions. We can predict model behavior under counterfactual input features by pushing model activations along feature directions. In the case of Othello-GPT, we can linearly decode information about the state of the board from activations – namely, which tiles are occupied by the current player’s pieces as opposed to the opponent’s pieces, and which tiles are empty. The fact that this information is linearly decodable in a given layer makes it easy to manipulate with simple vector arithmetic by pushing activations along feature directions. It should be noted that the linear representation hypothesis can be spelled out in slightly different ways, depending on how one technically defines what it means for a concept or feature to be linearly represented in the network [76]. In addition, there is room for disagreement about the theoretical grounding of this hypothesis. While linear decodability does make interpretability easier, this does not entail that complex neural networks do not also encode information non-linearly and cannot make sure of such information in downstream processing.

In Part I, we defined world models in the context of LLMs as internal representations of the world that allows them to parse and generate language that is consistent with real-world knowledge and dynamics [24]. Studies on Othello-GPT provide tentative evidence that Transformer models can acquire world models in this sense, at least in toy domains. For example, similar results have been found by studying Chess-GPT, a Transformer model trained on chess moves [77]. While Othello-GPT is not a language model properly speaking, it generates legal game moves in written form, and interventionist experiments reveal that it generates such moves on the basis of linearly decodable representations of the board state. To what extent can we generalize these findings to actual LLMs? This is a challenging question. A reasonable assumption is that Othello-GPT acquires a world model – an internal representation of the game world – because this is useful to improve its predictions of legal moves past a certain threshold. However, neural networks are notorious in their ability to achieve high accuracy scores on problems by mastering surface statistics; they tend to learn shortcuts (e.g., shallow heuristics) to achieve good performance on their learning objective until they hit a bottleneck. Furthermore, acquiring world models about the real world, rather than an extremely simple game world, is presumably costly in terms of training dynamics; it would require a substantial reorganization of the model’s internal structure, akin to the phase transition undergone by [15]’s model to ‘grok’ modular addition, albeit on a much broader scale.

It may well be that LLMs do undergo such phase transitions during training, and acquire representations akin to world models at least in some limited domains. This might be required to unlock certain abilities such as commonsense reasoning about intuitive physics, which in turn would further reduce the loss (i.e., improve next-token prediction performance) in certain contexts. For this to happen, two conditions must presumably be satisfied: (a) the text-based training data should provide enough information to induce the relevant components of world models; (b) the learning pressure to reduce loss on the next-token prediction objective should be sufficient to push the model to acquire such representations given long enough training trajectories. Neither condition is antecedently guaranteed, and there is currently limited evidence that they apply to most existing LLMs. In particular, interventionist evidence regarding the most capable LLMs such as GPT-4, whose behavior on certain tasks is most consistent with the acquisition of world models, is still sorely lacking. This is partly due to the fact that the weights of state-of-the-art LLMs are not publicly released, which precludes interpretability research from anyone other than researchers in the team that trains them. We will come back to this issue in section 3.3. As far as behavioral evidence is concerned, lingering failure modes on out-of-distribution tasks cast doubt on the hypothesis that current LLMs acquire sophisticated and robust world models [78, 79].

There are nonetheless ongoing efforts to assess the existence of world models in LLMs using the tools of mechanistic interpretability. For example, [19] used probing and intervention methods on models fine-tuned on the Alchemy and TextWorld datasets. Alchemy contains sequences of instructions for manipulating colored liquids in beakers (i.e., instructions for performing fictional chemical experiments and their outcomes), while TextWorld consists of textual transcripts of navigation in simulated worlds. Li et al. designed probes to test if the models’ contextual token embeddings encode and track the state of entities mentioned in the discourse. For example, in Alchemy the probe tries to determine if a representation encodes that a beaker is empty after its contents are drained. Li et al. used these probes to test whether intervening on the decoded entity representations would change model behavior. Specifically, they constructed two discourses \( x_1\) and \( x_2\) that describe draining liquid from two different beakers, \( b_1\) and \( b_2\) , resulting in one empty beaker per discourse. After encoding each discourse, they created a synthetic representation \( C_{mix}\) by taking the encoding \( C_1\) of discourse \( x_1\) and replacing the vector representations corresponding to the initial description of beaker \( b_2\) with those from \( C_2\) . Although \( C_{mix}\) does not correspond to any real textual input, it implicitly represents a situation in which both \( b_1\) and \( b_2\) are empty. When generating text conditioned on \( C_{mix}\) , they found that the model generates instructions that are more often consistent with both beakers being empty compared to generating from \( C_1\) or \( C_2\) alone. However, the generated instructions from \( C_{mix}\) are still not always fully consistent with the implicit state, suggesting the induced representation is approximate rather than perfect. By editing the entity representations to model a new state not seen in the actual training data or prompt, and observing changes in the model’s outputs that are often consistent with this new state, Li et al. provided tentative evidence that the model can induce an approximate implicit representation of the state of discourse entities purely from text. Note that producing text that is consistent with the hypothesis-driven manipulation of the LLMs activation space is not something that the system was trained to do. Furthermore, the method was found to be robust against ‘sanity checks’ to confirm that the intervention itself is not wholly responsible for injecting appropriate information into the system, such as finding negative results when attempting the intervention method on similarly-complex Transformer architecture with randomized weights. However, the generality of these findings remains uncertain, as the experiments focused on narrow domains with simple objects and dynamics. More systematic testing would be needed to determine how well they generalize to more complex and open-ended settings.

The project of mechanistic interpretability can be seen through the lens of causal abstraction [71]. Causal abstraction is a theoretical framework that aims to provide an interpretable high-level causal explanation of the behavior of a complex system, such as a neural network, that is consistent with the low-level causal structure of that system.

The core intuition is that the variables of a low-level causal model can be clustered into sets and aligned with the variables of a high-level causal model. The high-level model provides an abstract characterization of the low-level model if aligned high-level and low-level variables have equivalent causal roles and information content. This equivalence is experimentally verified with interventions on both levels that produce the same counterfactual behavior. More formally, an alignment between a low-level model \( \mathcal{L}\) and high-level model \( \mathcal{H}\) is a partition of the low-level variables into sets, with each set aligned to a high-level variable. A translation function maps low-level variable values to high-level values [80]. The alignment is causally consistent if, for any low-level intervention that has a corresponding high-level intervention under the translation function, intervening on both models produces equivalent results after translation. If an alignment is causally consistent, then \( \mathcal{H}\) is said to be a constructive causal abstraction of \( \mathcal{L}\) .

Any program or algorithm can be represented as a causal model [81]. In the context of interpretability research in deep learning, the low-level causal model is a neural network, containing many interconnected nodes and weights that determine its function. The high-level causal model proposed by mechanistic interpretability researchers offer hypotheses about the abstract computations and algorithms implemented by the network. Aligning groups of neurons and weights to single variables in the high-level model allows interpreting their collective causal role. The causal consistency condition ensures the high-level causal model faithfully captures the causal mechanisms embodied by the low-level neural network model. Causal abstraction thus enables the development of human-interpretable high-level causal models that accurately explain the reasoning and computations inside opaque neural networks.

A key method for assessing causal abstraction is interchange interventions, where neural representations created for one input are swapped into the model when it is processing another input. If the low-level neural model and high-level algorithm have the same counterfactual behavior under aligned interchange interventions, that provides evidence that the alignment witnesses a causal abstraction. Many of the methods described above, such as activation patching, can be used as interchange interventions. In fact, iterative nullspace projection can also be formalized within a causal abstraction framework [80]. Thus, causal abstraction is a useful framework to unify interventionist approaches to interpretability.

In practice, the project of mechanistic interpretability with large neural networks like LLMs can be seen as aiming for approximate causal abstraction, which relaxes the criteria for alignment [82]. The degree of abstraction can be quantified as the proportion of aligned interchange interventions that have equivalent effects in the high-level causal model and low-level causal model or target neural network [80]. When interchange intervention accuracy is 100%, the high-level model exactly abstracts the neural network. Otherwise, the high-level model approximately abstracts the neural network approximately, to the degree quantified by interchange intervention accuracy. Approximate abstraction can provide an interpretable high-level explanation that still reflects the causal structure of the target neural network with a degree of faithfulness suitable for genuine explanation.

Even if the preceding evidence regarding the complex and relevant functional roles attributed to LLM activation vectors are accepted, readers may be frustrated at the Transformer architecture’s apparent lack of biological inspiration or plausibility. Whereas other deep learning architectures like DCNNs for computer vision were substantially inspired by biological investigations into cortical tissues and have since even been further aligned with data from functional neuroscience (Buckner 2019), Transformers arose almost independently of the neuroscience of language processing. Moreover, the key-query-value division on which self-attention depends was inspired by an analogy to database theory in computer science rather than theory in psychology or cognitive science. What, we might wonder, could possibly correspond to the mathematical operations of self-attention in the brain, even at a gross functional grain of analysis?

Nevertheless, numerous imaging and alignment studies have suggested that some families of Transformers are also excellent predictors of activation patterns in human language processing areas, such as [20, 21]. Such alignment is defended even by researchers who concede that state-of-the-art Transformers like GPT-4 nevertheless lack substantial amounts of grounded world knowledge and other kinds of understanding often attributed to language comprehension, such as of complex semantic intentions or discourse representations. This has led some researchers to argue that such alignment demonstrates that left temporal areas of the brain associated with linguistic processing may play a limited word prediction role, arguing for a narrower construal of the language faculty than has traditionally attributed to such areas in cognitive neuroscience [83]. Others have been led to draw even more ambitious comparisons between the architecture of self-attention and the brain; for example, [22] have argued that operations hypothesized to be implemented by medial temporal lobe tissues in computational neuroscience – specifically, place and grid cell coding between the entorhinal cortex and hippocampus – can be seen as mathematically equivalent to self-attention operations in Transformers. Whether this abstract mathematical equivalence supports closer mechanistic alignment between Transformers and human neural tissues remains to be seen, but may yet be seen to soften some of the initial skepticism that such alignment is outright implausible.

LLMs exhibit impressive performance on traditional natural language processing tasks, where they have increasingly superseded task-specific models. However, there is only so much one can expect from a system exclusively trained to predict sequences of linguistic tokens. One promising idea to expand the capacities of LLMs is to augment or integrate them with components subserving other basic faculties – such as perception, imagination, and planning. There are two ways to execute this vision. The first involves modifying the Transformer-based architecture of traditional LLMs to allow them to process inputs in multiple modalities (e.g., text and images). The second involves incorporating LLMs as modules within a composite system, which may include multiple neural networks as well as classical symbolic algorithms. Each of these strategies has their own benefits and tradeoffs, but they are not mutually exclusive; a vision-language model designed to process images and text can itself be integrated with other module.

Language modeling plays a key role in these experiments. When Transformer architectures are augmented to receive visual input in addition to linguistic input, for example, their abilities to parse images is critically informed by linguistic concepts and dependencies induced from next-token prediction. By contrast, when traditional text-only LLMs are integrated within an ensemble of models, language acts as a common ground or ‘universal API’ for these models to communicate with each other without sharing differentiable parameters [84]. An API (Application Programming Interface) is a set of rules and protocols that allows different software components to communicate and interact with each other. By analogy, language can serve as a ‘universal API’ for generative models by providing a common interface to exchange information without being part of the same neural network. In either case, language modeling forms the backbone of more capable systems.

Efforts to move beyond text-only LLMs have converged in around two trends in deep learning research: multimodality and so-called ‘agent’ systems. In the rest of this section, we will discuss each of these trends and the philosophical questions they raise.

3.1.1 Multimodality

3.1.2 ‘Agent’ systems

Influential theorists like [25] have speculated that integrating Transformer-based language models in composite systems can provide other benefits to other modules. For example, they speculate that common causal variables tend to correspond to words in natural language, and so a language model that biased an image classifying module or a reinforcement-trained agent module would be more likely to focus on causally-relevant features of its training set (which they summarize with the hypothesis “that semantic variables are also causal variables”, [ 25, p. 14]). These variables include words referring to agents or subjects of sentences, actions (often captured by verbs), objects of agents’ actions (often captured by noun phrases in the direct object position), and modalities or properties of objects (often captured by adjectives and adverbs). From this perspective, natural languages are a kind of cultural technology that help us focus on the most important aspects of our environment [cf. 103], and systems that draw on language models to label incoming sensor data and formulate plans of action would thus be more likely to focus on causally relevant and robust properties in the environment and less likely to focus on artifacts and chimeras.

Many such architectures are already being implemented by industry and academic research groups, often inspired by the old Vygotskyan hypothesis in cognitive science that simulated inner speech is a powerful medium of thought that reliably develops in humans between the ages of 3-6, when children begin to demonstrate distinctively human success on problems of causal, logical, and social inference [104, 105, 106, 107]. To give an early example of such a system, Google Robotics has implemented an ‘inner monologue’ agent [108] that integrates a scene descriptor, success detector, and planning module together with a human interlocutor in a closed feedback loop. This agent can use a language model to interpret instructions given to it by a human, label visual input from sensors, formulate internal plans in terms of a constrained action language, and verify the success or failure of plan components by deploying a success detector on sensor input. Even more ambitious agent architectures deploying internal monologues are possible, such as using text encodings to drive internal simulations of expected perceptual output, which could then be used for long-term planning and counterfactual ‘imagination’ of possible scenarios – such as text-driven planning rollouts of the sort seen in DeepMind’s Imagination-Augmented Agents (I2A) architecture [109, 110].

These developments can be seen as specific steps on a broader journey from narrowly language-focused Transformer models to multi-modal Transformer-based ‘agents’, which many theorists see as the natural next step in deep-learning-based artificial intelligence. According to this frame, LLMs were an important breakthrough in making DNNs more compositional and in particular allowing them to tackle language-scaffolded effects in thought and action, but they are only one component in a full agent architecture. Moreover, the standard context of deployment for LLMs – continual next-word completion – lacks the stability and impetus of rational agency. This paradigm requires massive, ecologically-invalid datasets or at least deliberately-harvested instruction-tuning or reinforcement learning, and standard LLMs cannot gather their own ongoing training data by interacting with the world directly. Those trying to build Transformer-based agents seek to address these shortcomings by embedding LLMs in broader architectures and decision loops that allow them to generate their own goals (so-called ‘autotelic’ agents, [158]), formulate plans to pursue them, verify their success using a variety of modalities, and learn from continual self-driven exploration of a world.

There are three different kinds of ‘agent’ systems that build on the success of standard LLMs: (a) language agents that make use of external databases or tools; (b) embodied models that control a robotic body from natural language instructions with an LLM; and (c) multimodal models trained end-to-end to process actions in addition to text (Fig. 7).

Pan/Zoom

Language agents are systems with specialized modules orchestrated by calls a LLM (see [159] for a review). These modules may include a memory component, a planning component, and an action component. The memory component extends the input context window of the LLM, which acts as a short-term memory buffer, with an external long-term storage (like a text file or a structured database). For example, Reflexion combines a sliding context window with persistent memory banks [111]. The LLM can write to and read from the external memory, as well as summarize multiple memories into high-level insights to inform planning and action. A planning component can structure the reasoning process to accomplish goals. For example, ReAct utilizes action-observation cycles to refine plans based on environmental outcomes [112]. Finally, an action component can interpret the agent’s decisions into concrete outputs and behaviors.

A striking example of this approach is the ‘generative agents’ of [26]. The architecture has three main components: the memory stream stores a comprehensive natural language record of the agent’s experiences, with a retrieval model combining relevance, recency and importance to surface the most pertinent records; ‘reflection’ synthesizes memories into higher-level inferences that guide behavior over time; and planning translates conclusions from reflection and the current environment into high-level action plans that are recursively decomposed into detailed behaviors. Park et al. demonstrate the effectiveness of this approach in a Sims-like sandbox game environment called Smallville, inhabited by 25 unique language agents. These agents exhibit emergent social behaviors like information diffusion, relationship formation, and coordination (e.g. for spontaneously organizing a Valentine’s Day party), based solely on initial seed prompts. Another impressive example is Voyager, a language agent powered by GPT-4 that is able to continuously explore, develop skills, and make discoveries in the open-ended 3D world of Minecraft without any human intervention [113]. The key components of Voyager consist of an automatic curriculum module that proposes a stream of progressively more complex tasks optimized for exploration; a skill library for developing and composing complex, reusable behaviors; and an iterative prompting mechanism that leverages execution feedback and self-verification to iteratively refine GPT-4’s code generation until a task is successfully completed.

A concurrent trend consists in augmenting LLMs with external tools that can be manipulated through generated code or calls to APIs. One strategy consists in fine-tuning LLMs to enable them to learn in context when API calls are helpful in an entirely self-supervised manner, requiring only a handful of demonstrations per tool [114]. Augmenting LLMs with external storage and tools effectively creates hybrid neurosymbolic systems that use language and/or code as a universal interface between connectionist and symbolic components. In turn, this goes a long way towards addressing some remaining limitations of pure LLMs; for example, using tools such as a code interpreter or Wolfram Alpha significantly improves the performance of LLMs on challenging math problems [115, 116].

The second category of agent systems encompasses regular LLMs embedded in a system that include a physical robotic body roaming the real world. This approach combines the linguistic competence of LLMs with pre-trained robotic skills without requiring special modifications to the LLM component itself. For example, SayCan takes as input a high-level natural language instruction that describes a task for the robot to perform [117]. It then scores candidate low-level skills based on the probability that the skill’s description is relevant for the high-level goal – as determined by the LLM component – and the probability from a learned value function that the skill can successfully execute in the current state. By multiplying these probabilities, the method selects skills that are both useful for the goal and possible to achieve. The process repeats by appending selected skills until termination. SayCan can be implemented in a robot with vision-based manipulation skills trained via behavioral cloning and reinforcement learning, to perform real-world long-horizon tasks in a complex environment such as a kitchen.

The third category of agent system consists in Transformer-based models trained to receive multimodal inputs (including text) and able to produce actions as outputs. Unlike text-only LLMs embedded in robotic systems, these models are trained end-to-end to generate action commands. DeepMind’s Gato [118] is designed to be a ‘generalist agent’ that operates in a multi-modal, multi-task architecture on a generalist learning policy – which allows the same system to play Atari games, caption images, chat with humans, manipulate a real robot arm to stack blocks, and much more. In a similar spirit, [27] propose a method for incorporating Transformer-based vision-language models directly into robotic manipulation policies. The key idea is to represent robotic actions as special text tokens that can be generated as part of the model’s text output. The resulting model, RT-2, is pre-trained on image, text and action captioning datasets scraped from the internet, then fine-tuned on robotics datasets to output tokenized actions. The unified text-based output space of RT-2 is a key advantage over prior works, enabling it to successfully generalize to novel instructions after training.

The evolution of LLMs towards ‘agent’ systems capable of executing instructions, forming plans, and performing actions in the real world raises interesting philosophical questions. One pertains to the grounding problem: are embodied, LLM-based agents more likely to meet the requirements for referential grounding? [28] provide reasons to doubt that this is the case for systems, like SayCan, that simply make calls to a distinct text-based LLM component – where the latter is not pre-trained or fine-tuned on world-involving tasks. However, systems trained end-to-end to generate actions like Gato or RT-2 might deserve a different treatment. As Transformer-based models expand beyond text-only training to process any information that can be serialized – including images, videos, joint torques, button presses, etc. – it becomes less obvious that they should be deemed intrinsically incapable of inducing normative world-involving functions, even if they are not explicitly fine-tuned from human feedback. These systems also raise obvious question about agency itself: is the term ‘agent’ (let alone ‘autonomous agent’) a misnomer when it comes to systems that do not fundamentally have intrinsic goals or motivations? The answer largely depends on how thick one’s notion of agency is. LLM-based language agents are often described as generating, storing and retrieving goals, as well as breaking them down into sub-goals and individual actions. However, their behavior remains ultimately grounded in a human-written prompt. As such, they fail to meet the requirements for pure agential autonomy.

LLMs can eloquently converse about any topic sufficiently prevalent in their training data – including consciousness. Indeed, they are perfectly capable of generating compelling first-person reports that look like self-ascriptions of subjective experiences. In humans, such reports are generally taken at face value; while introspection can be unreliable [119], verbal reports typically provide (defeasible) evidence about someone’s experience. This intuition breaks down when it comes to LLMs. No matter how convincing their fluent mimicry of experience reports might be, it cannot be taken as prima facie evidence that they are conscious. Imitating patterns of human language use is precisely what LLMs are trained to do. Nonetheless, if LLMs can acquire emergent capabilities in the process of learning from a next-token prediction objective, then we ought to wonder whether current or future systems with a similar architecture could, in principle, acquire the capacity for conscious experience. Like other debates about the putative cognitive capacities of LLMs, this issue cannot be settled merely through armchair speculation, but calls for careful empirical research informed by mature theories.

While consciousness may seem like a particularly high bar to clear for machines that still clearly lack some of the hallmarks of human intelligence, it is worth nothing that it need not correlate with sophisticated cognitive abilities. Indeed, there is a rather broad consensus that many non-human animal species that presumably lack such abilities have the capacity for consciousness – not only mammals [120], but more largely vertebrates including reptiles, birds, and fish [121, 122], and perhaps even invertebrates including mollusks and arthropods [123, 124]. Unlike animals, however, AI systems do not share a common evolutionary history or physical substrate with humans. As such, considerations about phylogeny or similar neurophysiological features that are central to the discussion of animal consciousness do not straightforwardly apply to LLMs.

The hypothesis that artificial neural networks running on silicon chips could be conscious is premised upon computational functionalism – the view that what makes a physical system conscious is not its particular physical makeup, but whether it implements an appropriate set of computations [125]. If computational functionalism is true, biological brains and computers could in principle implement the same consciousness-enabling computations using different physical machinery. This assumption is not uncontroversial. Biological nervous systems exhibit complex dynamics involving global electrical patterns that modulate fine-grained neural signaling. Biochemical mechanisms and metabolic processes in the brain may impose specific constraints on the realization of conscious mental states [126, 127]. Artificial neural networks implemented on digital computing hardware bear little resemblance to biological nervous systems at this level of granularity, and it may be that the kind of silicon substrates used by modern computers cannot support consciousness even in principle.

If computational functionalism holds, however, then it should be possible in principle to identify the computational correlates of consciousness in biological brains, and assess whether similar computations are implemented in systems like LLMs. Given the lack of a complete and widely accepted scientific theory of consciousness, there is currently no definitive test or criteria to determine if an AI system is conscious. It is nonetheless possible to make qualified claims about the plausibility of consciousness in AI systems based on computational markers of biological consciousness derived from cognitive neuroscience research [128, 29, 129].

[29] survey prominent neuroscientific theories of consciousness that are compatible with computational functionalism. From these theories, they derive a list of ‘indicator properties’ – features or mechanisms that the theories associate with consciousness. The more indicator properties an AI system exhibits, the more likely it is to be conscious. Prominent theories of consciousness include, among others: recurrent processing theory, which proposes that recurrence within neural networks distinguishes conscious from unconscious processing [130]; global workspace theory, which claims that consciousness arises when information is broadcast to widespread networks from a limited-capacity workspace [131, 132]; and higher-order theories, which hold that consciousness depends on higher-order representation of lower-level activity [133, 134].

The authors cite evidence that current DNN architectures, including Transformer-based LLMs, fail to satisfy key indicator properties of consciousness associated with these theories. For example, the global workspace theory of consciousness requires the existence of modules operating in parallel, feeding into and taking inputs from a global workspace with limited capacity to create an information bottleneck. It is tempting to view the residual stream of Transformer-based LLMs as loosely analogous to a global workspace that ‘broadcasts’ information to dedicated attention modules. However, the bottleneck requirement is arguably not satisfied: the residual stream has the same dimensionality from input to output, and attention heads work with much lower-dimensional spaces. In addition, there is no genuine recurrent processing in a single forward pass through the model: an attention head in one layer can only store information for downstream layers to process, as opposed to making it globally available to all other attention heads.

This example highlights the challenge of reaching strong conclusions about the computations that current model architectures may or may not implement to vindicate claims about whether they satisfy ‘indicator properties’ of consciousness. Attention heads in Transformers do compress information in subspaces of the residual stream that are globally available for downstream layers to read from, albeit only within the forward pass. However, some variations of autoregressive Transformers enable recurrent processing that might, in principle, satisfy at least some of the core computational requirements identified by leading theories of consciousness [135, 136, 137]. Ultimately, as [29] note, most of the indicator properties extracted form these theories could likely be implemented in AI systems using current methods and architectural tweaks, although no existing system seems a strong candidate for consciousness in this respect. As with the assessment of other psychological capacities, more work is required to bridge low-level descriptions of algorithmic circuits in LLMs and high-level descriptions of their architectural features and behavior with intermediate-level descriptions of functional building blocks.

Much like the brains of humans and animals, LLMs today are large, opaque, and produce strikingly sophisticated behavior; this has led some researchers to suggest that we need a new ‘science of machine behavior’ to develop appropriate instruments and methods of analysis to study them [138]. In developing these methods, we should be careful to import general lessons that were hard won in other sciences of behavior, such as human and comparative psychology. As mentioned in the companion paper, these sciences have from the beginning been fraught with biases like anthropomorphism and anthropofabulation, and as a result have hit numerous speed bumps in their history. For example, early comparative psychology relied too much on the method of anecdotes, which led to numerous dead ends and overinterpretation of behaviors which might have simply been statistical flukes [139]. This led modern comparative psychology to emphasize the need for reproducible experimental conditions on model organisms that could be raised in controlled conditions for all labs. Several areas of social, comparative, and developmental psychology are today also facing ‘replication crises’, where well-known psychological effects and mechanisms that were thought to be well-confirmed and taught routinely in textbooks have been found to be based on statistical artifacts that were reified through questionable methodological practices, such as p-hacking or outright data fraud [140, 141]. This has similarly led to calls for pre-registration of all experimental protocols, training stimuli, datasets, and analysis methods to be published alongside with experimental results, so that other researchers can confirm findings on their own and scrutinize experimental materials directly to explain any discrepancies [142, 141].

Unfortunately, research on LLMs seems set to repeat and even exacerbate these problematic practices from the history of psychology. Most of the highest-performing LLMs are the results of enormous investments in high-quality datasets for training, labor-intensive fine-tuning, and proprietary architecture tweaks. These datasets and training protocols are regarded as valuable trade secrets that are closely guarded by companies like Google and OpenAI who seek to control market share by outperforming their competitors. As a result, many of the most alluring behavioral results, such as those reported in [157], are essentially anecdotes about systems which are unavailable to outside researchers (especially by interventionist methods) and by design irreproducible by other research groups. For commercial reasons, the results are frequently hyped on social media and in press releases, often beyond the scientific merit of the underlying findings. Even before the advent of LLMs, methodologically-focused researchers worried about the lack of reproducibility in deep learning research [143], which has only been exacerbated in the age of LLMs. Most academic research has been conducted on models that are publicly available for academics such as BERT, which has led to the field being metonymically dubbed ‘BERTology’ [144]. Prominent researchers on LLMs have already called for disregarding analyses based on anecdotal evidence and urged for early adoption of best practices for reproducibility in architecture, data, training methods, and analyses [145, 146]. There have also been developments of large, collaborative, open-access benchmarking methods such as BIG-bench to evaluate LLMs in a more controlled and objective manner [147]. At the same time, empirical results may suggest that certain discontinuous gains in performance in LLMs can only be achieved at extremes of scale and computation power [148, 149]. If explainability and safety research is to address the largest systems which are likely to be deployed by large technology companies like OpenAI, Google, and Microsoft, researchers cannot naively hope that research focused on smaller open-access models will always generalize. Future researchers will have to learn how to strike a balance between the scientific need for reproducibility and the practical need to understand the latest achievements of large, proprietary, closed research efforts.

A refrain that has recently appeared in many popular science articles, blogs, and social media posts holds that LLMs may reflect the discovery of an entirely new kind of ‘alien’ intelligence that is fundamentally unlike our own. This may be seen as a mere statement of fact – an optimistic appraisal that the linguistic behavior already exhibited by state-of-the-art LLMs is obviously intelligent, coupled with a pessimistic appraisal that the underlying processes producing this behavior are much at all like ours. With respect to most of the philosophical questions raised above, however, it reflects an almost complete dodge and a return to a naive behaviorism about intelligence by use of an eye-catching label. In any particular case, we need to ensure that before we accept the use of such a label as a descriptor of some scientifically-important LLM behavior that we have scientifically-respectable methods to assess that behavior along the important dimensions reviewed in Part I.

As [30] points out, this language first began to appear as a descriptor of deep learning performance well before the current boom in LLMs, especially regarding new machine-learning-based methods to analyze data in high-complexity sciences like biology and fundamental physics. There, the language of ‘alien’ intelligence was conceptually tied to older philosophical dreams of rationalists like Descartes to create a science that was free from the shackles of human perspective and bias – an objective science free from the foibles of human ‘powers and faculties’, as Hume put it. The hope kindled by ‘alien intelligence’ here is that by automating the process of data collection and analysis and relaxing requirements on transparency and intelligibility, “nonhuman sensors and neutral algorithms can replace these human ‘powers and faculties’ ” with alien agents that are wholly “data-driven and increasingly are able to construct their own models and search strategies”. Indeed, there is some reason to think that breakthrough systems like AlphaFold, which have nearly solved the highly-complex protein folding problem that eluded human microbiologists for more than a century, achieved their results precisely by discovering inscrutable features that are too complex for humans to understand (Buckner 2020, Kieval 2023). Whether one buys the talk of alien intelligence and inscrutable features in this context, however, the concept derives its utility from pragmatic scientific goals that humans largely accept – the ability to predict and perhaps control natural phenomena.

When this notion of ‘alien intelligence’ is applied to the linguistic behavior of LLMs, however, the standards relevant to assessing its utility are far less clear. We suggested in Part I that there are many reasons to be skeptical of purely behavioral criteria for intelligence when applied to products of modern LLMs, for they may in any instance be mere Blockheads that, like a student who has merely memorized the answer key, are simply regurgitating information contained in their training sets. A purely behavioral criteria for intelligence that makes no mention of learning efficiency or representational properties should also be rejected, for it gives us no way to say that two systems that display the same behavior differ in intelligence, even when they use very different procedures to achieve the same goal. We also reviewed reasons in section 2 above to be skeptical that implicit assumptions on which benchmarks or other measures are based could be trusted, without evidence of deeper mechanistic or algorithmic correspondences with human subjects. As such, we should resist the application of ‘alien intelligence’ to LLMs until this term can be explicated in a way that addresses these concerns and outlines a scientifically-responsible set of practices for evaluating internal performance differences amongst various LLMs.

Summarizing the previous discussion leads to the following conclusions. Firstly, DNNs like Transformers have have an astounding capacity for memorization and learn from enormous training sets. As a result, any particular behavior they exhibit could be mere regurgitation or approximate retrieval, rather than demonstrating human-like processing or understanding of the task. This is what we characterized as our null hypothesis in Part I [24]. Secondly, however, not all LLM behaviors can be explained by this null hypothesis. Interventionist methods suggest that, at least in some cases, their behavior is caused by robust representations of task variables.

Thirdly, these models do not yet generalize as well out of distribution as humans or some classical models. Like many other ANNs trained by gradient descent, LLMs exhibit varying degrees of abstraction in their generalization behavior, depending on the inductive biases of their neural network architecture and the exploitable informational structure of their training data. They may exhibit weak systematicity or islands of strongly systematic behavior, but not fully classical systematic generalization to very different task stimuli. While some ‘non-content-specific’ processing does occur in LLMs [12], their behavior still falls short of the full consistency and generality exhibited by humans.

This limitation is likely due to the fact that most LLMs only model formal aspects of human linguistic competence [83]. Their designers make little attempt to model a human-like cognitive architecture, which might include multiple semi-independent modules corresponding to domain-general psychological faculties that enable and maintain consistency in humans – such as perception, memory, imagination, attention, social cognition, metacognition, reflection, and long-term planning. Without these additional components, LLMs remain limited in their ability to match human-level systematicity across diverse contexts. The lack of such a comprehensive agent architecture probably also partly explains why LLMs learn so much less efficiently than humans, requiring orders of magnitude more training data before achieving comparable performance.

Considering available evidence, a middle-ground position emerges regarding the relevance of LLMs to human cognition. Current-generation LLMs may serve as inefficient but useful tools for reverse-engineering partial models of the processes or algorithms that humans use to generate linguistic behavior [155]. While it would be far-fetched to consider LLMs adequate computational models of the human mind, let alone human learning, they are not entirely irrelevant to human cognition and intelligence.

LLMs may capture partial models of human cognition by learning representations and computations similar to those used by humans, at least in specific domains. These learned representations and computations can be flexibly combined using operations on continuous vectors, enabling LLMs to generate novel behaviors. The resulting behaviors can reproduce not only the surface-level linguistic patterns of human language use but also, to a limited extent, the underlying systematicity and generalization ability characteristic of human cognition. It is worth noting that no previous AI technology has been able to achieve this level of fidelity to human linguistic and cognitive behavior. However, it is also important to acknowledge that LLMs often still fall short of human efficiency and flexibility when generalizing to novel stimuli.

To be clear, even this middle-ground position remains speculative. We have reviewed evidence from intervention methods suggesting that LLMs can represent syntactic and semantic properties of their utterances, but the degree of success achieved by current interventions is difficult to interpret as definitive. Weaker evidence from interchange interventions can often be obtained for alternative representational interpretations of networks, and these various interpretations may be mutually exclusive. Simply regarding the strongest interpretation as the ‘correct’ representational interpretation of a trained LLM requires further justification.

This issue reflects a more general fact about language and language use: philosophers of language and linguists have long noted that words and sentences in typical dialogues are inherently ambiguous and vague. It may be that only when sentences are produced and controlled by agents with stable viewpoints and communicative intentions do they acquire determinate meanings. Current generations of LLMs likely lack such agential stability, as readers can verify for themselves by observing the malleability of their responses. It is easy to prompt the current generation of chatbots to recant previously affirmed statements and defend inconsistent positions by simply suggesting that they are wrong, confused, or offensive. In short, while LLMs may accumulate information about the same syntactic and semantic properties as humans and even combine that information in flexible ways to create novel outputs, they nevertheless lack the agential stability required to imbue their utterances with determinate meanings that remain stable over time.