Based off the paper 
arXiv.org Probing Classifiers: Promises, Shortcomings, and Advances
Part 1: Introduction
We hear it all the time: a language model "learns" or "knows" things. But what does that actually mean? When an LLM correctly answers that the capital of France is Paris, how is that fact-the relationship between the subject "France," the relation "is the capital of," and the object "Paris"-stored inside billions of parameters?
Is it smeared across the entire network, like a drop of ink in a pool of water? Or can we point to a specific set of neurons, a specific layer, and say "Aha! This is where the model thinks about capitals!"?
We could see the model's final output, but the internal process was a complete "black box." This is a massive problem, especially for AI safety. If we don't understand how a model "thinks," how can we ever trust it? How can we prevent it from acting on hidden biases or reaching dangerous conclusions based on flawed internal reasoning?
This is the puzzle that drives interpretability research. We need tools to peek inside the black box. And one of the first, most fundamental tools researchers developed is called probing. It’s our first step in moving from just using models to actually understanding them.
The Probing Framework: A Simple Question for a Complex Mind
So, how can we start to map the inside of a model? The core idea behind probing is simple. If you think a certain kind of information (like grammar, or facts) is stored in a specific place in the model, you can try to "read" it from that location.
Think of it like this: You have a giant, complex alien machine (the LLM), and you suspect a certain cluster of blinking lights represents the machine's knowledge of verbs. You can't understand the alien wiring directly. So what do you do?
You build a simple, human-made sensor (the probe). This sensor does only one thing: its light turns green if it detects a verb, and red otherwise. You then point your sensor at that cluster of alien lights and feed the machine thousands of words. If your sensor's light blinks green every time a verb passes through-and only when a verb passes through-you've found something! You've "probed" the machine and confirmed that, yes, this part of the machine tracks verbs.
That, in a nutshell, is the entire conceptual framework. We train a small, simple classifier-the "probe"-on the internal activations (the "blinking lights") of a larger, frozen model.
Let's break down the technical steps:
- Choose a Model and a Property: First, we pick our target. This could be any pre-trained model like BERT or GPT-2. Then, we decide on the linguistic property we want to hunt for. Are we looking for part-of-speech tags? Syntactic tree structures? Factual knowledge?
- Extract the Representations: We feed a large dataset of text into the model. For every word or sentence, we save the internal state, or "representation," from a specific layer. These are just vectors-long lists of numbers-that represent the model's "thoughts" about that piece of text at that specific point in its computation.
- Train the Probe: This is the key step. We take these saved representations and use them as features to train a simple classifier (often just a basic linear model). The goal of this probe is to predict the property we're interested in. For example, can it look at the vector from Layer 8 for the word "running" and predict that it's a "verb"?
Crucially, during this process, the giant language model is kept completely frozen. We don't change its weights at all. Why? Because we're not trying to teach the big model anything. We are performing an act of scientific measurement. The probe is just our measurement tool, our "sensor," for reading what's already there. - Evaluate the Probe's Performance: Finally, we see how well our probe does. If a super-simple linear probe can predict part-of-speech tags with 95% accuracy just by looking at the representations from Layer 8, that's powerful evidence. It tells us that the information about "verb-ness" or "noun-ness" isn't just vaguely present; it's explicitly and linearly separable in that part of the model. Voilà! We've localized a piece of knowledge.
This framework gives us a systematic way to create a "map" of the model's brain, layer by layer, showing us where different types of knowledge reside. It was a foundational breakthrough, turning the black box into something we could finally start to chart. But as we'll see in the next section, this simple picture comes with some very tricky complications.
Part 2: The Trouble with Probes
In Part 1, we laid out the elegant and simple framework of probing. It felt like we had found a key to the black box-a way to systematically map the internal knowledge of a language model. It's an exciting idea! But, as with most things in science, the simple picture is never the full story.
The moment researchers started publishing their results, a healthy and absolutely critical debate began. The core of it was this: how do we know what our probe is really telling us? This led to a series of crucial challenges that have made the field of interpretability so much more rigorous.
Shortcoming #1: Is 87.8% a Good Score? The Need for Controls
Imagine we run our experiment. We train a probe on Layer 8 of BERT to find part-of-speech tags, and we get an accuracy of 87.8%. Fantastic, right?
Well, maybe. But what does 87.8% actually mean? Is that a high number or a low number? Without context, a number is meaningless. This is the first major problem: we need comparisons and controls.
- Baselines: What's the dumbest possible score? We could compare our 87.8% to a majority baseline (what if we just guessed "noun" every time?). More cleverly, what if we trained the probe on the outputs of a randomly initialized, untrained model? Some studies did this and found-shockingly-that even random features can give you a surprisingly decent score! WOW. This tells us that our 87.8% has to be significantly better than the score from random noise to mean anything.
- The Big One: Is the Probe a Thermometer or a Calculator?
This is the question at the heart of the probing debate, raised by researchers Hewitt and Liang (2019). Is our probe a simple thermometer, passively reading the information that's already clearly represented in the model? Or is it a powerful calculator in its own right, learning to compute the linguistic property from scratch using the complex representations as its raw input?
Think about it. The probe is a neural network, even if it's a small one. What if it's just really smart? What if it's not finding knowledge, but creating it by learning complex patterns from the rich vector representations?
To solve this, they introduced the genius idea of a control task. Here's how it works:
The central debate in probing: Is our probe a simple 'thermometer,' passively reading information that's already there? Or is it a powerful 'calculator,' learning the task from scratch and creating the knowledge itself? This single question changed the field - You run your normal probing experiment. Let's say you get 87.8% accuracy.
- Then, you take your dataset and randomly shuffle all the labels. So "running" is now labeled "noun," "table" is labeled "adverb," and so on. It's complete nonsense.
- You train a new probe on this nonsensical, randomized data.
- Now, if your probe was a simple thermometer, its accuracy on this nonsense task should be terrible, near random chance. But if the probe is a powerful calculator, it might be able to memorize the random associations for the words in the training set, achieving a surprisingly high score.
This gives us a new, much more meaningful metric: selectivity.
Selectivity = (Accuracy on Real Task) - (Accuracy on Control Task)
A high selectivity score tells you that your probe is genuinely finding structure that exists in the model's representations, not just memorizing the probing task itself. This is why many researchers now argue for using very simple, linear probes. A complex, multi-layer probe might get a higher accuracy score, but it's much more likely to be a "calculator," leading to low selectivity and misleading you about what the base model truly knows.
Shortcoming #2: The Elephant in the Room - Correlation vs. Causation
Okay, so let's say we've done everything right. We've used a simple linear probe, we've run control tasks, and we have a sky-high selectivity score. We can now confidently say that information about part-of-speech is clearly represented in Layer 8.
But here's the billion-dollar question: Does the model actually use this information to do its job?
This is the classic problem of correlation vs. causation. We've found a correlation-the presence of grammatical information in the representations. But we haven't proven that the model relies on this information to, say, predict the next word.
Think about the way the probing framework is set up. We train the big LLM first. Then we freeze it. Then, in a completely separate step, we train our little probe on its frozen representations. The main model and the probe never interact. The probe is just an observer looking at a static snapshot.
This is a huge limitation. Maybe the information is just an accidental byproduct of the model's main training process. Maybe it's there, but the model completely ignores it and uses other, more bizarre correlations to make its predictions.
How do we get around this? This is where the cutting edge of research is heading. Instead of just passively observing, researchers are now developing interventional methods. What if we could perform "brain surgery" on the model?
For example, researchers are now trying to:
- Ablate Information: Use techniques to actively erase the part-of-speech information from the representations in Layer 8 and then see if the model's performance on its main task (like language modeling) gets worse. If it does, that's strong evidence for a causal link!
- Modify Representations: Use the probe to find the "part-of-speech" direction in the activation space and then manually shift the representation along that axis. Does this change the model's output in predictable ways?
These interventional approaches are far more complex, but they are the necessary next step to move beyond just finding correlations and toward understanding the true, causal computations happening inside these models. It's a fundamental shift from map-making to real reverse-engineering.
Shortcoming #3: The Goldilocks Problem - Which Probe is "Just Right"?
So we've decided to build our "sensor" to check the model's internal state. What should that sensor be made of? Should it be a simple magnifying glass or a high-powered electron microscope? This is the debate over the complexity of the probe itself.
- The Argument for Simplicity (Linear Probes): Many researchers advocate for using the simplest tool possible, like a basic linear classifier. The philosophy is beautiful: if a dead-simple probe can find the information, then that information must be "lying on the surface." It must be explicitly and easily accessible in the representations. This gives us high confidence that we're not fooling ourselves, and it helps us avoid the "probe as a calculator" problem we discussed last time.
- The Argument for Complexity (MLP Probes): But what if the information isn't lying on the surface? What if it's there, but it's tangled up and encoded in a complex, non-linear way? A simple linear probe would fail and we'd mistakenly conclude the information isn't there at all. Other researchers argue we should use a more powerful, complex probe (like a multi-layer perceptron or MLP) to give us the best possible estimate of all the information that could theoretically be extracted.
So, who's right? It turns out, there's a trade-off. A more complex probe might give you higher accuracy, but as we saw with the "selectivity" metric, that accuracy might be an illusion because the probe itself is doing all the heavy lifting.
This has led to a more nuanced view. Instead of just reporting accuracy, researchers now talk about the accuracy-complexity trade-off. Some even use sophisticated ideas like Minimum Description Length (MDL). You can think of MDL like this: it doesn't just ask "Did the probe get the right answer?" but rather, "How much effort did the probe have to expend to get the right answer?" A probe that gets high accuracy with very little effort (a low MDL score) is much more convincing. It tells us the information was easy to find.
This whole debate shows us that choosing a probe isn't just a technical decision; it's a statement about what you're trying to measure. Are you looking for easily accessible information, or are you trying to find any trace of the information, no matter how encoded?
Shortcoming #4: The Streetlight Effect - Are We Only Looking Where It's Easy?
This final set of problems is perhaps the most profound because it's about the limitations of us, the human researchers.
- Datasets aren't Tasks: We talk about probing for a "task" like "grammar," but in reality, we're using a finite, messy "dataset" as a proxy. The choice of that dataset profoundly impacts our results. If you train a probe on part-of-speech tags from the Wall Street Journal, your conclusions might not apply to a model's understanding of poetry. Even worse, the original data the LLM was trained on is a huge confounding variable. We're often not comparing architectures; we're comparing models trained on different slices of the internet, and we have no way of untangling the two.
- We Can Only Find What We Look For: This is the big one. The entire probing framework relies on us having a pre-defined, pre-labeled property z to look for. We can probe for part-of-speech because linguists have already given us the labels. We can probe for syntax because we have annotated treebanks.
But what if a language model, in its alien mind, discovers a new, powerful, and entirely non-human way of representing language? What if it learns a concept that is incredibly useful for predicting the next word but doesn't map to any existing human linguistic category?
We would never find it. Because we would never know to look for it.
This is the ultimate Streetlight Effect: we are searching for our lost keys only under the streetlight, because that's where the light is. Probing is limited to the concepts we can already name and annotate. The model's most interesting discoveries might be lurking just outside the lamplight, in the darkness, and our current methods give us no way to see them.
Conclusion: Probing is Flawed, Foundational, and Forward-Looking
So, after all these critiques, is probing a useless tool?
Absolutely not. Probing is one of the most important ideas in the history of interpretability. Its initial, simple promise-that we can map the knowledge in a model-forced the field to get serious. The intense debate about its shortcomings is a sign of a healthy, maturing science.
Probing taught us to be more rigorous. It forced us to invent concepts like control tasks and selectivity. It made us confront the monumental gap between correlation and causation, pushing the community toward more powerful interventional methods. And it made us humbly acknowledge the limits of our own hypotheses.
It's the first rung on the ladder of understanding. We may not use probes to get the final answer, but we use the lessons learned from probing every single day. The black box is still dark, but probing was the first time we figured out how to switch on a flashlight, even if it could only illuminate a small corner at a time. And that changed everything.
A note on illustrations: All conceptual diagrams in this post were generated by the author using prompts for Google's Gemini model.