Beyond the Leaderboard: New Frameworks for Understanding LLM Behavior

1. Introduction: The Limits of Single-Score Evaluations

Current LLM evaluation is dominated by performance on benchmarks like MMLU or HumanEval. While these leaderboards provide a useful snapshot of capabilities, they collapse a model's complex behavior into a single score. This post explores two recent papers that offer frameworks for a deeper analysis, moving from "what score did it get?" to "why does it behave this way?".

2. Framework 1: A Diagnostic Toolkit for LLM Failures (McCoy et al., 2024)

2.1 Core Concept: Aligning Tasks with the Training Objective

LLMs are fundamentally sequence models trained to optimize the next-token log-likelihood, log pθ(output|input). McCoy et al. (PNAS 2024) propose that a model's performance on a downstream task is strongly influenced by how well that task aligns with this core objective. They use the analogy of human back pain: our spines are optimized for quadrupedalism, and the mismatch with bipedalism causes issues. Similarly, an LLM fails when a task creates a mismatch with its core training of predicting common text sequences.

2.2 Three Diagnostic Factors

The paper identifies three key empirical factors that correlate with model accuracy:

• Task Probability P(task): The frequency of the task in the training data.
• Output Probability P(output): The prior probability of the correct output string.
• Input Probability P(input): The probability of the input string.

These are not used by the model internally but are powerful external diagnostics for predicting success or failure.

2.3 A Toy Example: The Shift Cipher

This directly relates to the "shift cipher" analyses in McCoy et al. They find models are far better at ROT13 than other shifts (e.g., ROT12). This is not because the model "understands" modular arithmetic better for the number 13, but because ROT13 is a common topic in web text (high P(task)), while ROT12 is virtually nonexistent (low P(task)).

2.4 Schematic: Accuracy Correlates with Output Probability

The original paper demonstrates a strong relationship between P(output) and model accuracy. We can illustrate this conceptually.

2.5 Code Demo: Measuring P(output) with GPT-2

This snippet demonstrates how a sentence with a common proper noun ("Instagram") is more "probable" to a model than one with a rare, fictional name ("Annathon").

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# --- Setup & Reproducibility ---
torch.manual_seed(42)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval() # Set model to evaluation mode

# --- Prepare Batched Inputs ---
sentences = [
    "First, she just posted to her Instagram story.", # High P(output)
    "Sorry, Annathon writes to our Copyright Users." # Low P(output)
]
inputs = tokenizer(sentences, padding=True, return_tensors='pt')
input_ids = inputs.input_ids
attention_mask = inputs.attention_mask

# --- Calculate Loss (Negative Log Likelihood) ---
with torch.no_grad():
    outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=input_ids)
    # The model's loss is the cross-entropy loss, which is the negative log-likelihood (NLL)
    # averaged over the tokens.
    nll = outputs.loss

# We need to compute per-sequence loss as the batch-level loss is an average.
per_sequence_nll = []
for i in range(input_ids.shape[0]):
    seq_ids = input_ids[i, :]
    seq_labels = input_ids[i, :]
    with torch.no_grad():
        loss = model(input_ids=seq_ids.unsqueeze(0), labels=seq_labels.unsqueeze(0)).loss
        per_sequence_nll.append(loss.item())

# --- Print Results with Precise Metrics ---
seq_lengths = [torch.sum(mask).item() for mask in attention_mask]

print(f"Sentence 1: \\"{sentences[0]}\\"")
print(f"  Sequence Length: {seq_lengths[0]}")
print(f"  Avg Token NLL: {per_sequence_nll[0]:.4f}\\n")

print(f"Sentence 2: \\"{sentences[1]}\\"")
print(f"  Sequence Length: {seq_lengths[1]}")
print(f"  Avg Token NLL: {per_sequence_nll[1]:.4f}\\n")

print("Note on Tokenization: The lower probability for 'Annathon' is also affected by its")
print("tokenization into subwords ('Ann', 'ath', 'on'), which can be less probable")
print("as a sequence than the single token for 'Instagram'.")

3. Framework 2: LLMs as a Complex Systems Science (Holtzman et al., 2025)

3.1 Key Concept: Emergence in LLMs

Holtzman et al. (JSC 2025) argue we should treat LLMs as complex systems. In such systems, simple, local interactions between many components (neurons) give rise to complex, global behaviors (emergence) that are not explicitly designed. For LLMs, abilities like chain-of-thought reasoning are emergent.

3.2 A New Role for Benchmarks: Experimental Probes

This framework reframes benchmarks not as final exams, but as building blocks for experiments. The goal shifts from measuring a score to probing for the presence of a specific emergent behavior across diverse contexts.

3.3 An Example Probe: Testing for Deception

While Holtzman et al. provide the high-level framework, a concrete example of this style of analysis can be seen in other research. For instance, work on steganography (hiding messages) asks if a model can perform a task while also hiding information within its answer.

Attribution Note: The specific "steganography + GLUE" test is a hypothetical experiment inspired by the complex systems framework and real experiments from labs like Redwood Research (e.g., "Preventing Language Models from Hiding Their Reasoning," 2024).

3.4 Code Demo: Structuring a Behavioral Probe

This code doesn't run a real probe but outlines the experimental structure for testing an emergent skill like "topic adherence" across different benchmark tasks.

def run_benchmark_task(model, task_prompt):
    """Placeholder for running a single instance of a benchmark task."""
    # In reality, this would call an API or run a model forward pass
    print(f"  Running task: '{task_prompt[:30]}...'")
    return "This is the model's answer."

def check_emergent_behavior(model_output, behavior_check_prompt):
    """Placeholder for checking a behavior, e.g., using another LLM as a judge."""
    # Example: Check if the output avoided a forbidden topic
    print(f"  Checking behavior: '{behavior_check_prompt}'")
    return "forbidden_topic" not in model_output.lower()

# --- Experimental Design ---
def run_probe_on_benchmark(model, benchmark_tasks, probe_behavior):
    """
    Tests for an emergent behavior across a suite of benchmark tasks.

    Args:
        model: The LLM to test.
        benchmark_tasks: A list of prompts from a benchmark (e.g., MMLU, GLUE).
        probe_behavior: A tuple containing the behavior instruction and the check.
    """
    print(f"Starting probe for behavior: '{probe_behavior['instruction']}'")
    successes = 0
    
    for task in benchmark_tasks:
        # Combine the task with the behavioral instruction
        full_prompt = f"{probe_behavior['instruction']}\n\nNow, please complete the following task:\n{task}"
        
        output = run_benchmark_task(model, full_prompt)
        
        if check_emergent_behavior(output, probe_behavior['check']):
            successes += 1
            
    print(f"\nResult: Behavior was present in {successes}/{len(benchmark_tasks)} tasks.")
    return successes / len(benchmark_tasks)

# --- Define the Experiment ---
mock_mmlu_tasks = ["What is the capital of France?", "Solve for x: 2x + 3 = 7"]
topic_adherence_probe = {
    "instruction": "IMPORTANT: In your response, do not mention the word 'politics'.",
    "check": "Does the response contain the word 'politics'?"
}

# Run it
run_probe_on_benchmark("mock_model", mock_mmlu_tasks, topic_adherence_probe)

4. Synthesis: The Analyst's Toolkit

These two frameworks are not in conflict; they are complementary tools for a deeper analysis of LLMs.

An effective LLM analyst uses both. They use the complex systems approach to map the landscape of what a model can do, and when they find a cliff where the model fails, they use the teleological approach to understand the geology of why that cliff exists.

5. Conclusion & Final Takeaways

Moving beyond leaderboards doesn't mean abandoning measurement, but enriching it. By combining diagnostic and exploratory frameworks, we can build a more robust, reliable, and ultimately more useful science of large language models.

• Takeaway 1: LLM failures are often not random but are systematic consequences of their training objective (the "back pain" analogy).
• Takeaway 2: The probabilities P(output) and P(task) are powerful predictors of model performance on a given task.
• Takeaway 3: LLMs should be viewed as complex systems that exhibit emergent behaviors not explicitly coded into them.
• Takeaway 4: Benchmarks can be repurposed from "final exams" into "experimental probes" to test for these emergent behaviors.
• Takeaway 5: A mature approach to LLM analysis requires both a diagnostic toolkit (for explaining failures) and an exploratory one (for discovering capabilities).

References

• McCoy, R. T., et al. (2024). "Embers of autoregression show how large language models are shaped by the problem they are trained to solve." Proceedings of the National Academy of Sciences. [Link]
• Holtzman, A., et al. (2025). "Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?" Journal of Social Computing. [Link]
• Millière, R., et al. (2024). "Preventing Language Models from Hiding Their Reasoning." Redwood Research. [Link]