Part 4 Reverse Engineering Transformers: Induction Heads

💡

Final part in the series:
Part 1: Reverse Engineering Transformers: Deconstructing Attention

Part 2: Reverse Engineering Transformers: Attention Heads & Circuits
Part 3 Reverse Engineering Transformers: Path Expansion

Transformer Circuits VI: Induction Heads

TL;DR

• Induction heads are a two-head circuit that implements a simple but powerful form of in-context learning: repeating sequences.
• The circuit's algorithm is: "find a previous occurrence of the current token, attend to the token after it, and copy that token's identity to the current position."
• This is implemented via K-composition with a previous token head from an earlier layer.
• The previous token head shifts information, making a token's Key vector at position t encode information about token[t-1].
• The induction head's Query vector at position t+k can then match with the Key at position t, allowing it to attend to t and use its OV-circuit to copy token[t].
• The discovery of induction heads shows that powerful, general capabilities like in-context learning can emerge from simple, mechanically understandable circuits.

Context & Motivation

One of the most remarkable abilities of large language models is in-context learning: the ability to perform a task from a few examples in a prompt, without any weight updates. If you show a model A -> B, C -> D, E ->, it will likely predict F. How does it do this?

The authors of the "Mathematical Framework" paper discovered that even tiny, two-layer models develop a surprisingly general version of this skill. When trained on random text, they learn to complete repeating sequences (e.g., ...[A][B]...[A] -> [B]). The circuit responsible for this is the induction head.

Understanding this circuit is a major milestone. It's not just a description of a pattern; it's a full, end-to-end reverse engineering of a key algorithmic capability. It demonstrates how the abstract building blocks we've discussed assemble into something greater than the sum of its parts.

Prereqs

• A complete understanding of Parts I-V, especially head composition and the roles of W_Q, W_K, and W_V.

Core Idea in One Picture

The induction circuit consists of (at least) two heads. A "previous token head" in Layer 0 sets up the information, and the "induction head" in Layer 1 executes the core logic.

graph TD
    subgraph Input Sequence
        direction LR
        A1("... token[t-1]") --> B1("token[t]") --> C1("...") --> A2("token[t+k-1]") --> B2("token[t+k]") --> Next("???")
    end

    subgraph Layer 0: Prev Token Head (H0)
        B1 -- Attends to --> A1
        A1 -- Copies info --> B1_Stream[Stream at B1]
    end

    subgraph Layer 1: Induction Head (H1)
        B2 -- Query: "I am B" --> B1_Stream
        B1_Stream -- Key: "Prev token was A" --> B2
        B1_Stream -- Value: "I am B" --> B2_Stream[Stream at B2]

        B2 -- Attends to --> B1
        B1 --OV Circuit --> B2_Stream
    end

    B2_Stream -- Projects via W_U --> Logit_for_C["'C' logit is high"]

    style H0 fill:#ccf
    style H1 fill:#fcc
    style B1_Stream stroke:#00f,stroke-width:2px,stroke-dasharray: 2 2

Figure 1: The Induction Head circuit. At a token B (at t+k), the model wants to predict the next token. If another B appeared at t, the circuit finds it. It does this because a Layer 0 head (H0) copied information about A (token[t-1]) into the residual stream at B's position (t). H1 at t+k can now attend to B at t (by matching its Query for 'B' with 'B's content) and use its Value to copy the next token's identity, thereby predicting 'C'.

Walkthrough: The Algorithm Step-by-Step

Let's trace the full algorithm for the sequence ...[A] [B] [C]... [A] [B] -> ?, where the model should predict [C]. We are at position t+k, which contains the token [B].

Step 1: The Setup (Previous Token Head, H0)

• Head Type: A "previous token head" in an early layer (let's say Layer 0).
• QK-Circuit: This head has a very simple QK circuit. Its weights are configured such that the query at position t will always give the highest attention score to the key at position t-1.
• OV-Circuit: This head's OV circuit is a "copying" circuit. It takes the embedding of the previous token (token[t-1]) and adds it to the residual stream at position t.
• Result: After Layer 0, the residual stream at every position t, x_t, contains a representation of both token[t] (from the original embedding) and token[t-1] (added by H0).
  • At position t, where the token is [B], the stream x_t now effectively says: "I am [B], and the token before me was [A]."

Step 2: The Core Logic (Induction Head, H1)

Now we move to a later layer (Layer 1). The induction head H1 executes the main algorithm.

• Where does H1 attend? H1 needs to attend from the current position t+k (token [B]) back to the previous occurrence of [B] at position t.
• The QK-Circuit (K-Composition in action):
  • Query: The query vector q_{t+k} for H1 at the current position is formed from the stream, so it says "I am [B]."
  • Key: This is the clever part. The key vector k_t at the earlier position t is formed from its stream x_t. And x_t contains information about token[t-1], which is [A]. So, k_t says something like "The token after me is [B]" or "My predecessor was [A]".
  • The induction head's W_Q and W_K matrices are learned to match these specific signals. The query q_{t+k} looks for a key k_t that corresponds to a previous occurrence of the same token. The K-composition (the effect of H0 on H1's key) makes this possible. The Query "I am [B]" matches the Key "The token after me is [B]".
• What does H1 move? The OV-Circuit:
  • H1's attention is now focused on position t.
  • Its W_V matrix is trained to read the information from the residual stream at t, x_t. Crucially, x_t still contains the original embedding for token[t], which is [B].
  • However, the composed W_V^1 W_O^1 circuit of the induction head performs a specific transformation: it is trained to take the an embedding from the stream and effectively "shift it forward by one". It acts to copy the next token.
  • Wait, how can it copy the next token, [C], by looking at [B]? The paper hypothesizes that the model leverages existing correlations. Since [B] is often followed by [C] in the training data, the model can learn a W_V W_O transformation that maps the embedding of [B] to a vector that boosts the logit for [C].

Self-correction/Refined View: A simpler and more common view of the induction head's OV-circuit is that it attends to the position t-1 (the position of A) and copies the value from there. The logic is:

Let's stick to the paper's primary finding: the QK circuit attends to the previous instance of the same token, and the OV circuit copies the next token. The key is that W_Q and W_K are trained to "look for [token]" and H0's output lets them look for "[token] in the next position".

Implications & Limits

• Implication: This is a concrete mechanism for in-context learning. Many of the impressive few-shot learning capabilities of LLMs may be powered by scaled-up, more abstract versions of this basic circuit.
• Implication: It shows that general, algorithmic capabilities can emerge spontaneously from a simple predictive loss, and that these emergent algorithms can be reverse-engineered.
• Limit: This is one specific circuit. It doesn't explain all forms of in-context learning (e.g., following complex instructions). However, it provides a foundational example.
• Limit: The explanation relies on the linear approximation provided by path expansion; the real dynamics with LayerNorm are noisier and more complex, but the core signal is captured by this model.

Takeaways

• Induction heads are a two-layer, two-head circuit that allows a model to continue repeating sequences.
• They are a mechanistic explanation for a simple form of in-context learning.
• The circuit combines a "previous token head" (to provide history) and an "induction head" (to use that history).
• K-Composition is the critical mechanism that allows the induction head's query to find a match based on the previous token's identity.
• The induction head's QK-circuit locates the previous occurrence of the current token.
• Its OV-circuit acts to copy the token that followed it.
• The existence of such a clean, effective, and discoverable circuit provides strong evidence for the validity of the circuits research agenda.

Transformer Circuits VII: A Conclusion and The Road Ahead

TL;DR

• This series reverse-engineered Transformers using the "circuits" framework, moving from high-level architecture to specific, learned algorithms.
• The core idea is to treat a trained model as a program to be decompiled, not an inscrutable black box.
• We built a hierarchy of abstractions: the residual stream (the memory), independent heads (the operators), QK/OV circuits (the sub-routines), and composition (the program flow).
• This framework allowed us to fully dissect the induction head circuit, a concrete mechanism for in-context learning.
• Major frontiers remain: understanding FFN/MLP layers, tackling non-linearities like LayerNorm, dealing with superposition, and scaling these analyses to frontier models.

Context & Motivation

Over the last six posts, we've journeyed deep into the internals of Transformers. We began with a single, ambitious goal: to move beyond treating models as black boxes and instead reverse engineer the algorithms they learn during training. The "Mathematical Framework for Transformer Circuits" provided our roadmap.

We started with the broadest architectural concept-the residual stream-and progressively zoomed in, dissecting layers into heads, heads into sub-circuits, and finally, tracing the compositional paths that weave these components into complex programs.

This final post zooms back out. We will recap our journey, solidify the key insights of the circuits perspective, and honestly assess its limitations and the exciting frontiers that lie ahead for the field of mechanistic interpretability.

The Journey from Architecture to Algorithm: A Recap

Our entire analysis was built on a series of increasingly granular, yet complementary, mental models.

This final step is the proof-of-concept for the entire framework. It demonstrates that our abstractions are not just neat metaphors; they are precise enough to fully reverse-engineer a key capability of the model.

Implications & The Bigger Picture

The circuits perspective is more than just a set of analytical tricks; it's a paradigm for studying neural networks. Its core implications are:

• Models Learn Interpretable Algorithms: The most profound takeaway is that models don't just learn a soup of statistical correlations. They appear to learn clean, specific, and often human-understandable algorithms made of composable parts.
• A Path to Rigorous Safety and Auditing: If we can truly understand the circuits driving a model's capabilities, we can begin to audit them for flaws, biases, or unintended behaviors. This is a far more robust approach to safety than relying on behavioral testing alone.
• A New Kind of Neuroscience: By analogy, MI is like a form of computational neuroscience for artificial minds. We are moving from lesion studies (ablations) to the detailed tracing of neural pathways to understand how cognition emerges from component interactions.

Limitations and the Road Ahead

For all its successes, the circuits framework is still in its infancy, and major challenges remain.

Final Takeaways

• The "circuits" perspective provides a powerful, hierarchical framework for reverse-engineering Transformers.
• The core mental model is of components reading from and writing additive updates to a central residual stream.
• This framework allows us to decompose the model into understandable parts: layers, heads, and QK/OV sub-circuits.
• Depth enables composition, where components chain together to form multi-step algorithms like induction heads.
• Induction heads provide a concrete, mechanistic explanation for an important form of in-context learning.
• While powerful, this linear, attention-focused view is an approximation that must be expanded to handle non-linearities (LayerNorm), FFN layers, and superposition.
• Mechanistic interpretability is a nascent but promising field that aims to make AI models truly understandable, not just superficially predictable.

This journey from architecture to algorithm is just the beginning. The tools and concepts laid out here are the foundation upon which the next generation of interpretability research is being built, aiming to one day make even the most complex AI systems transparent and trustworthy.

Further Reading