Inner Workings of a Transformer: Interpretability Techniques

This post is a detailed walkthrough of two key families of interpretability techniques-Behavior Localization and Information Decoding-as presented in Sections 3 and 4 of the paper. We explore how to pinpoint which parts of a Transformer are responsible for a given behavior and how to understand the information those parts represent.

• The Key is the Architecture: The Transformer's core design-summing component outputs into a residual stream and then linearly projecting that sum to get logits-is not just an implementation detail. It's the mathematical foundation that makes principled attribution and intervention possible.
• Localization (Sec. 3): Techniques like Direct Logit Attribution (DLA) and activation patching directly exploit the model's additive structure. DLA accounts for how much each component pushed the final answer, while patching causally tests a component's importance by swapping its output.
• Decoding (Sec. 4): Techniques like probing, linear feature analysis, and Sparse Autoencoders (SAEs) investigate what information is encoded in the model's activations. These methods help us build hypotheses about what concepts the model has learned, which can then be causally verified with localization tools.
• A Unified Workflow: We will frame these two sections as a coherent workflow: start with the model's linear structure, use localization to find responsible components, use decoding to hypothesize what they represent, and iterate to build a grounded understanding of the model's internal algorithm.

Bridge from Section 2: Why This All Works

Before diving into specific techniques, we must ground our analysis in the mathematical structure of the Transformer, as laid out in Section 2 of the paper. At its heart, a Transformer is a sequence of layers, each containing attention and MLP components. Crucially, each of these components, which we can index with c, computes an update vector h_c and adds it to the residual stream.

Let r^(l) be the state of the residual stream at a specific token position after layer l. The final residual stream state, r_L, is the sum of the initial token/positional embedding and all subsequent component updates:
r_L = r^(0) + h_{attn,0} + h_{mlp,0} + ... + h_{attn,L-1} + h_{mlp,L-1}

For simplicity, we can express this as a sum over all components c in the computational graph:
r_L = ∑_c h_c

The model then produces logits, z, by applying a final linear transformation, the unembedding matrix W_U. The logit for a specific token y, denoted z_y, is the dot product of the final residual stream with the corresponding token's unembedding vector, u_y:
z_y = u_y^T r_L = u_y^T (∑_c h_c)

By the linearity of the dot product, this expands to:
z_y = ∑_c u_y^T h_c

This simple equation is the bedrock of modern mechanistic interpretability. It tells us that the final logit for any token is just a linear sum of contributions from every component in the model. This structure licenses two powerful modes of analysis :

With this architectural foundation in mind, we can now explore the techniques in Sections 3 and 4 as principled methods for exploiting this structure.

Section 3: Behavior Localization - Finding Which Parts Matter

Behavior localization aims to identify which parts of the model-from individual input tokens to specific attention heads or MLP neurons-are responsible for a given output. These methods are our primary tools for moving from "the model predicted token Y" to "the model predicted token Y because head 7.3 attended to token X and FFN 12 modified the representation in this way."

3.1. Input Attribution: Tracing Behavior Back to Tokens

The most basic localization question is: which input tokens were most influential for a given prediction?

Gradient-based Attribution:
A standard approach in machine learning is to use gradients. If we have a function f(x) that maps an input x to a scalar output of interest (like the logit z_y), its first-order Taylor expansion f(x + Δ) ≈ f(x) + ∇_x f(x)^T Δ tells us that the gradient ∇_x f(x) represents the sensitivity of the output to infinitesimal perturbations of the input. We can aggregate these sensitivities per input token embedding to get a saliency score.

A common variant is gradient·input, which computes ∇_x f(x) ⊙ x, where ⊙ is the element-wise product. This approximates the contribution of each input feature to the final output score. However, raw gradients can be noisy and suffer from saturation. Techniques like SmoothGrad (averaging gradients over noisy inputs) and Integrated Gradients (integrating gradients along a path from a baseline input to the actual input) provide more robust estimates by mitigating local noise and accounting for non-linearities along the integration path .

Intuition: Gradient attribution asks, "If I could wiggle each input token embedding a tiny bit, which wiggles would have the biggest impact on the final logit?" It's a measure of local sensitivity.

Attention Weights vs. Context Mixing:
It is tempting to use attention weights as a direct measure of input importance. However, this is often misleading. An attention head could place a high weight on a token, but if the value vector produced at that token position is near-zero or orthogonal to what the rest of the model cares about, its actual influence is negligible. The weights only tell you where the information is being routed from, not what that information is or how it's used .

More faithful methods account for the full information flow through the attention head's OV-circuit (the part that computes Attention(Q,K)V). This can involve weighting attention scores by the norm of the value vectors or using more complex rollout algorithms that recursively trace influence through multiple layers. However, simple rollout techniques can misattribute importance by assuming linearity where it doesn't hold .

Contrastive Attribution:
Models don't just predict a token y in a vacuum; they predict it instead of other tokens. The softmax function amplifies differences between logits. Therefore, a more meaningful attribution target is often the logit difference, z_y - z_y~, where y~ is a plausible alternative token. By attributing this difference, we ask, "Which inputs caused the model to favor y over y~?" This aligns our analysis with the competitive nature of the model's output layer.

Caveat: Input attribution methods, especially gradient-based ones, provide a correlational signal, not a strictly causal one. They can be insensitive to causal model components, disagree with each other, and fail when perturbations push the model into out-of-distribution territory where its behavior is unpredictable. They are best used as a first-pass tool for hypothesis generation .

3.2. Model Component Importance: Pinpointing Heads and Neurons

While input attribution is useful, the core of mechanistic interpretability lies in understanding the internal components. Here, we leverage the Section 2 decomposition to its fullest.

Direct Logit Attribution (DLA):
As derived earlier, the total logit z_y is a sum of contributions from all components: z_y = ∑_c u_y^T h_c. The term u_y^T h_c is the Direct Logit Attribution (DLA) of component c to token y. It is the projection of the component's output vector h_c onto the unembedding direction u_y for that token. A large positive DLA means the component directly "pushed up" the logit for y.

Just as with input attribution, a contrastive analysis is often more powerful. The Contrastive Direct Logit Attribution for y vs. y~ is:
DLDA_c(y, y~) = u_y^T h_c - u_y~^T h_c = (u_y - u_y~)^T h_c

This measures how much component c pushed the logits in favor of y over y~. Because this method is a direct calculation based on the model's forward pass, it is computationally cheap and provides a complete, additive decomposition of the logits .

Intuition: DLA is the ultimate accounting tool. It opens up the final sum ∑_c h_c and shows you exactly how much each h_c term contributed to the final answer along the u_y direction.

Causal Interventions via Activation Patching:
DLA tells us about correlations in a single forward pass. To establish causality, we need to perform interventions. Activation patching is the canonical technique for this. The methodology is as follows:

The core intervention is a <- a~, where a is the activation on the corrupted run and a~ is from the clean run. A large Δ on a metric S means the component is causally important for that metric.

Caveat: The choice of corrupted/counterfactual input is critical and can significantly affect the results. An overly simplistic ablation (e.g., zero-ablating a component) might push downstream components into an out-of-distribution regime, causing them to behave erratically. This concept, termed ecological validity, emphasizes the importance of using counterfactuals that keep the model in-distribution . Moreover, models can exhibit self-repair, where downstream components compensate for a patched or ablated component, masking its true importance.

Subspace Activation Patching and Distributed Interchange Intervention (DII):
Often, a feature or concept is not represented by a single neuron but by a direction or subspace within the high-dimensional activation space. Instead of patching an entire activation vector a, we can perform a more surgical intervention.

Given a subspace S (e.g., spanned by a single direction vector v), we can define a projector P_S onto that subspace. A subspace patch replaces only the part of the activation that lies in S:
a_patched = P_S a~ + (I - P_S) a

Here, we take the feature from the clean run (a~) and inject it into the corrupted run (a) while preserving everything in the orthogonal complement of S. This approach, also known as Distributed Interchange Intervention (DII), allows us to test the causal role of specific, distributed features rather than entire activation vectors . This is a powerful refinement of patching that aligns with the hypothesis that features are linear.

Finding Circuits: Edge and Path Patching:
Components don't just act independently; they form circuits. An early attention head might extract information that a later MLP processes. Edge patching extends activation patching to test the importance of the connection between two components. For example, to test the A -> B connection, one would patch the output of A only as it is read by B, leaving other components that read from A unaffected.

Path patching generalizes this to entire chains of components. However, the number of possible paths is combinatorially explosive. This has led to automated search algorithms like ACDC (Automatic Circuit Dissection and Correspondence) that greedily prune edges from the computational graph. Other methods like Edge Attribution Patching (EAP) use gradient-based approximations to estimate the effect of patching all possible edges more efficiently, though these linearized estimates can fail in the presence of strong non-linearities .

Section 4: Information Decoding - Discovering What is Represented

While Section 3 focuses on where a computation happens, Section 4 focuses on what information is being computed and represented. These techniques aim to put meaningful labels on the vectors and subspaces within the model.

4.1. Probing

A probe is a simple, supervised model (often linear) trained to predict a property of interest (e.g., part-of-speech, syntactic depth, presence of a specific concept) from a Transformer's internal activations. If a probe can be trained to high accuracy on the activations r^(l) at layer l, it suggests that information about that property is linearly decodable from the residual stream at that layer.

Caveat: This is a crucial distinction: a successful probe demonstrates that information is encoded, not necessarily that it is used by the model to solve its primary task. A powerful probe might simply be learning a complex function that the main model has no need for. To mitigate this, researchers use controls (e.g., training a probe on randomized labels to see what accuracy it can achieve by chance) and apply principles like Minimum Description Length (MDL) to penalize probe complexity, ensuring the discovered correlation is not spurious . A probe gives you a hypothesis; Section 3 tools are needed to test it.

4.2. The Linear Representation Hypothesis and Sparse Autoencoders (SAEs)

A central idea in modern interpretability is the linear representation hypothesis: many concepts and features the model learns are represented as directions or subspaces in activation space. This hypothesis is powerful because it connects directly to the linear readout mechanism from Section 2. If a concept C is represented by a direction v_C, then adding α * v_C to the residual stream should predictably shift the model's behavior towards expressing concept C.

This enables two key interventions:

• Erasure: To test if a feature is necessary for a behavior (e.g., stereotypical gender bias), we can identify its direction v and project it out of the residual stream: r_new = r - P_v r. If the behavior disappears, the feature was causally necessary.
• Steering: We can guide model generation at inference time by adding a feature vector to the residual stream, r_new = r + α * v, effectively "steering" the model towards a desired attribute or style.

The surprising effectiveness of these simple linear edits provides strong evidence for the linear representation hypothesis .

Unmixing Features with Sparse Autoencoders (SAEs):
A major challenge is that neurons are often polysemantic, meaning a single neuron activates for multiple, unrelated concepts. Furthermore, models may use superposition to represent more features than they have dimensions by storing them in a non-orthogonal basis.

Sparse Autoencoders (SAEs) are an unsupervised technique designed to tackle this. An SAE is a simple neural network trained to reconstruct a model's internal activations (e.g., r^(l)) from a much wider, "overcomplete" set of latent features. The key is a strong sparsity penalty (L1) on the latent activations. The objective is:
min_W,θ E[ ||r - W^T s||_2^2 + λ||s||_1 ]
where s = σ_θ(r) is the sparse code produced by the encoder, W is the decoder dictionary, and λ is the sparsity coefficient.

The columns of W represent learned "features." Because the code s is forced to be sparse, each activation r is reconstructed using only a few active features from this dictionary. These SAE features are often much more monosemantic (interpretable) than the original neuron basis. Once an SAE is trained, we can analyze its features using the full suite of Section 3 tools: calculate their DLA, patch them, and search for the inputs that maximally activate them .

Intuition: An SAE acts like a prism for activation vectors. It takes a dense, tangled vector where many features are superimposed and splits it into a sparse set of clean, interpretable feature directions.

Caveat: SAEs are not perfect. There is a trade-off between reconstruction fidelity and sparsity. Furthermore, the reconstruction error r - W^T s itself can contain important information, and causal analyses must account for it. Newer architectures like Gated SAEs improve this trade-off by separating the detection of a feature from its magnitude, leading to better fidelity for a given level of sparsity .

4.3. Decoding via the Vocabulary Interface

The unembedding matrix W_U is the bridge between the model's internal representation space and the human-readable vocabulary space. We can leverage this bridge to decode the content of internal states.

• Logit Lens: The simplest application is the logit lens, where we apply the unembedding matrix directly to an intermediate residual stream r^(l) to get a premature prediction: z^(l) = W_U^T r^(l). This can reveal how the model's prediction evolves layer by layer. However, raw logit lenses often produce poor results because the distribution of r^(l) can be very different from that of the final residual stream r_L. Tuned lenses learn a simple affine transformation to correct for this distributional shift .
• Patchscopes: This technique combines patching (Section 3) with a readout mechanism. The core idea is to patch an activation from a source context into a target context and then immediately decode it using a readout function (like the logit lens) to see what information it contains in that new context. This provides a more controlled way to understand what an activation vector "means" .
• Logit Spectroscopy and SVD Views: We can analyze the linear operators within the model (like FFN weight matrices) by understanding their effect on the vocabulary. By taking the Singular Value Decomposition (SVD) of a weight matrix, we can find the input directions it amplifies the most. Projecting these directions into vocabulary space via W_U reveals which tokens or concepts the matrix is most sensitive to. This "logit spectroscopy" helps characterize the function of entire MLP layers or attention heads .
• Maximally Activating Inputs: A straightforward way to understand a component (like a neuron or an SAE feature) is to search through a large dataset for the input examples that cause it to activate most strongly. This provides concrete examples of what the feature responds to. A riskier alternative is to use optimization to synthesize an input that maximizes the activation, but this can produce unnatural, "frankenstein" inputs that exploit adversarial patterns, leading to interpretability illusions .

Opinion: The ability of modern language models to generate their own natural language explanations for their features is tantalizing. However, as the primer correctly warns, these explanations should be treated as hypotheses, not ground truth. A model might generate a plausible-sounding but completely unfaithful explanation. Such claims must be rigorously tested using the causal intervention methods from Section 3.

Synthesis: A Coherent Workflow for Understanding Transformers

Sections 3 and 4 of the primer are not just a catalog of methods; they describe a unified, iterative process for reverse-engineering the algorithms learned by Transformers. The workflow, grounded in the Section 2 decomposition, looks like this:

This loop-from localization to decoding and back to causal verification-allows us to move from coarse-grained observations about model behavior to fine-grained, validated claims about the specific algorithms the model has learned.

Limits and Open Questions

Despite this progress, significant challenges remain. The primer highlights several critical limitations that practitioners must keep in mind:

• The Attention-as-Explanation Fallacy: Raw attention weights are not explanations. Always consider the full OV-circuit.
• Distribution Shift in Patching: Causal interventions can be misleading if they push the model into out-of-distribution states. Careful design of counterfactuals is paramount.
• Probe Faithfulness: Probes can discover spurious correlations. Their findings must be validated with causal methods.
• SAE Quality: The utility of SAEs depends on finding the right balance of sparsity and reconstruction fidelity, and their learned features are not guaranteed to be perfectly monosemantic.
• The Limits of Linearity: While the linear representation hypothesis has been incredibly fruitful, models are fundamentally non-linear. Linearized methods like EAP can fail, and complex reasoning may involve non-linear interactions between features that are difficult to analyze.