Training Data Attribution: Corroborative Methods

What is “Corroborative Attribution”?

Training Data Attribution (TDA) is a broad term for methods that connect a model's behavior back to its training data. Within this field, a critical distinction exists between two primary goals: corroboration and contribution.

Corroborative attribution aims to find evidence. Given a specific model behavior-such as generating a piece of text, making a factual claim, or exhibiting a toxic stereotype-the goal is to search a candidate corpus and identify examples that directly support, contain, or entail that behavior. The central question is one of provenance and support: “Can I find a document in this dataset that substantiates what the model is saying?”

This stands in contrast to contributive attribution (Training Data Attribution: Contributive Methods). Contributive methods, like influence functions or TracIn, seek to estimate the causal impact of a training example on a model's parameters or a specific prediction. Their question is counterfactual: “How much would this prediction have changed if the model had not been trained on this specific example?”

While contributive methods offer deep insights into model learning, they are often computationally expensive and require significant access to the model's internals (e.g., gradients, checkpoints), which is frequently infeasible for practitioners working with proprietary, third-party models.

Corroborative attribution, on the other hand, is a powerful and practical first-line-of-defense auditing tool. It enables:

• Data Provenance: Tracing model outputs to their likely sources for copyright, licensing, or fact-checking purposes.
• Leakage & Contamination Audits: Systematically checking if evaluation benchmarks or sensitive test sets are present in the training data, which would invalidate reported performance metrics.
• Safety Reviews: Identifying the sources of undesirable behaviors like toxicity, bias, or the memorization of personally identifiable information (PII).
• Policy & Deployment Decisions: Providing concrete evidence to inform decisions about model deployment, even when the model itself is a black box.

Background & Formalization

We can formalize the goal of corroborative attribution as follows. Let $B$ denote an observed model behavior, such as a generated sentence, a classification decision, or a factual claim. Let $C$ be a candidate corpus, which could be a known pre-training dataset (like The Pile) or a suspected source of information (like a crawl of a specific website).

A corroborative method, or a corroborator, is a process that returns a support set $S_{\text{targ}} \subseteq C$. For each example $z \in S_{\text{targ}}$, we say that $z$ supports the behavior $B$, which we denote as

$z \Rightarrow B$. This support relationship is established by an evidence relation $R$ and a threshold $\tau$. Formally:

$z \in S_{\text{targ}} \wedge R(z, B) \geq \tau \implies z \Rightarrow B$

The evidence relation $R(\cdot, \cdot)$ is the heart of the method and can take several forms, each with its own strengths and limitations:

Caveat: Corroboration is not proof of training. Finding a document $z$ in corpus $C$ that supports behavior $B$ demonstrates that the corpus contains information that could plausibly have led to that behavior. It does not definitively prove that the model used that specific document to produce the output. The same information could be present in many other documents or even in a different, unknown training corpus.

Paper 1 - What's In My Big Data? as a Corroborative Audit

The ICLR 2024 paper What's In My Big Data? (WIMBD) by Elazar et al. serves as a landmark example of a large-scale, systematic corroborative audit. Instead of focusing on a single model's behavior, the authors developed a platform to analyze the properties of the data itself, creating a foundation for countless future corroborative investigations.

Method: "Count + Search at Scale"

The core innovation of WIMBD is its ability to perform two fundamental operations-counting and searching-efficiently across massive text corpora. The authors analyzed ten major datasets, including C4, The Pile, RedPajama, and LAION-2B-en, totaling over 35 terabytes of text. By building specialized indices and using optimized search tools, they could execute complex queries on a standard compute node, making large-scale data audits accessible without requiring a supercomputer.

Key Findings as Corroborative Evidence

WIMBD's analyses directly enable corroborative workflows by characterizing the evidence available in these corpora. Their findings were striking:

• High Duplication: The study revealed massive levels of duplication. For example, they found that approximately 50% of the documents in both RedPajama and the English portion of LAION-2B are duplicates. This is a critical finding for attribution; a behavior corroborated by a document present thousands of times is more likely to be memorized or heavily weighted by a model.
• Benchmark Contamination: The authors systematically searched for test and evaluation data within the pre-training corpora. They found significant contamination for key benchmarks, including the Winograd Schema Challenge (WSC), and parts of the GLUE and SuperGLUE suites. This discovery corroborates the hypothesis that high performance on these benchmarks may be due to memorization rather than true generalization.
• PII and Toxicity Prevalence: The platform was used to search for patterns matching personally identifiable information (email addresses, phone numbers) and to score documents for toxicity. The widespread presence of this content provides the necessary substrate for corroborating model behaviors that leak PII or generate toxic language.

WIMBD is a quintessential example of corroborative auditing because it links potential model behaviors (high benchmark scores, PII leakage) directly to evidence at the corpus level. It does not measure the influence of any single document but rather characterizes the sea of data from which these behaviors could plausibly emerge.

Intuition: The analyses in WIMBD effectively transform the opaque concept of "pre-training data" into a searchable, queryable substrate. This move from an unknown blob of text to an indexed and analyzed corpus is the foundational step that makes corroborative attribution possible. It allows a researcher to move from "the model said something weird" to "the model said something weird, and here are 10,000 documents in its likely training data that say the same thing."

Paper 2 - Why Corroboration Matters (You Are What You Eat)

The position paper You Are What You Eat argues that a deep understanding of the relationship between training data, a model's internal structure, and its downstream generalization is necessary for AI alignment. It posits that merely evaluating a model's behavior on test sets is insufficient to guarantee safety, as different internal structures can produce identical outputs on a given dataset but generalize in dangerously different ways.

The central thesis is that data shapes structure, and structure determines behavior. This provides a powerful motivation for investing in robust corroborative audits. If we want to build safe, aligned AI, we cannot treat the training data as an afterthought. We must understand its contents, its flaws, and its latent structures, as these will inevitably be reflected in the models we train.

Corroborative attribution is the most direct method we have for drawing this line from data to behavior. When we corroborate a model's toxic output by finding toxic source documents, or corroborate a factual error by finding source documents containing the same misinformation, we are empirically verifying the "You Are What You Eat" principle.

Opinion: For practitioners working on the front lines of AI safety and deployment, especially with limited access to model internals, corroboration is often the only tractable audit lever available. While mechanistic interpretability and contributive attribution are vital research directions, the ability to quickly search a corpus for evidence supporting a harmful behavior is a practical, scalable, and indispensable tool for risk assessment today.

Method Families & Practical Protocols (Corroborative)

A practical corroborative audit follows a structured protocol, moving from a behavior of interest to a final report. This protocol typically combines several families of methods.

Method Families

A Practical Protocol

A typical corroborative audit workflow can be structured as follows:

Caveat: The efficacy of any corroborative audit is limited by the accessibility of the training corpus. When auditing a model trained on a closed-source, proprietary dataset, one must rely on open-source proxies (e.g., using RedPajama as a proxy for the Llama training set). In such cases, the report must clearly state this limitation and frame the findings as plausible evidence rather than definitive proof.

Evaluation & Reporting

A rigorous corroborative audit is not just about finding examples; it's about evaluating the process itself. If a ground-truth dataset exists (e.g., a set of known memorized sequences and their sources), the performance of a corroborator can be measured with standard metrics:

• Precision: What fraction of the retrieved support examples ($S_{\text{targ}}$) are truly relevant?
• Recall: What fraction of all possible relevant examples in the corpus $C$ did the method successfully retrieve?

In practice, practitioners should perform ablations over the similarity threshold $\tau$ to understand the trade-off between precision and recall. Reporting should be standardized and reproducible, including:

• The exact corpora searched, with versioning information.
• The query templates or construction methods used.
• The software and models used for embedding, search, and entailment.
• For audits involving human review, inter-annotator agreement scores should be reported to quantify the subjectivity of the task.

Tools like dataset cards and datasheets should be used not just for the training data itself, but for the outputs of corroborative audits, providing a transparent record of what was searched for and what was found.

My View

Opinion: Every high-stakes audit of a powerful language model should begin with a corroborative analysis. It is the most direct, scalable, and empirically grounded way to assess data-related risks. Only when this foundation is established, and when access permits, should teams move to more complex and computationally intensive contributive methods. Corroboration provides the "what" and "where"; contribution can then explore the "how much."

Intuition: The core argument of You Are What You Eat-that data shapes structure which in turn shapes behavior-is deeply compelling. If this causal chain holds, then examining the source data is not just one of many options; it is an indispensable first step. Ignoring the properties of the training data is akin to a doctor diagnosing a patient without asking about their diet.

Caveat: Corroborative methods have blind spots. They are biased towards finding evidence that is lexically or semantically close to the target behavior. They can over-credit popular sources that are highly duplicated and may fail to identify support from a collection of diverse, long-tail documents that, in aggregate, teach the model a concept through paraphrase and analogy. An audit report must always be transparent about the types of evidence it is designed to find and the types it is likely to miss.

Limits & Open Questions

Despite its practical utility, corroborative attribution faces several significant challenges and open research questions:

• Coverage vs. Privacy: How can we conduct thorough audits when training corpora are increasingly proprietary and closed-source? Can we develop methods for privacy-preserving corroboration that allow model developers to answer queries about their data without revealing it?
• Bridging to Contribution: Finding a supporting document is the first step. The next is understanding its impact. Future research should focus on integrating corroborative search (to find candidates) with lightweight contributive estimation (to rank candidates by influence).
• Watermarking and Fingerprinting: Could we proactively embed invisible signals (watermarks) or unique phrases (fingerprints) into training corpora? This would transform corroboration from a search problem into a simple detection problem, making attribution far more reliable.
• The Cost of Scale: While WIMBD demonstrated feasibility on a single node, analyzing future trillion-token datasets will require continued innovation in efficient indexing and search algorithms.

Ultimately, corroborative methods are a vital part of the toolkit for ensuring AI systems are safe, transparent, and accountable. They ground our understanding of model behavior in the concrete reality of the data they consume, reminding us that, in the world of AI, you are what you eat.

References