Training Data Attribution: Contributive Methods

The field's reliance on chasing leaderboard scores is slowly giving way to a more mature focus on model understanding, and for good reason: benchmark score-chasing isn’t enough; we need methods that link predictions to specific training examples. Without this connection, we are flying blind, unable to explain why a model fails on a critical edge case, reproduces a harmful bias, or succeeds at a surprising new task. Training Data Attribution (TDA) provides the analytical lens for forging this link.

What Counts as “Training Data Attribution”?

At its core, Training Data Attribution is the process of assigning a score to one or more training examples, z_i, that quantifies their responsibility for a model's behavior on a specific test point, z_test, or for its overall performance. It seeks to answer the question: "Which training data points made the model make this specific prediction?"

This is distinct from several related concepts. Data provenance simply tracks the origin of data, without assessing its impact on the model. Memorization detection identifies instances where the model has stored and reproduced training data verbatim, which is a specific type of influence but not the whole story. Dataset auditing typically involves high-level statistical analysis of a dataset (e.g., class balance, subgroup representation) rather than example-level attribution for a specific prediction.

The motivation for TDA has become increasingly urgent. Research such as "Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning" (Razeghi et al., 2022) demonstrates that a model's emergent abilities are deeply connected to the statistical properties of its training corpus. For example, they show that the frequency of numbers in the pretraining data correlates with a model's ability to reason about those numbers. While this provides a corpus-level explanation, TDA offers the tools to drill down and ask: given this statistical backdrop, which specific examples were most instrumental in teaching the model a given concept or causing a particular failure?

Background & Scope

Before diving into methods, let's establish our notation and scope. We consider a training set D = {z_i = (x_i, y_i)}_{i=1}^n of n examples. Our model is parameterized by θ ∈ R^p. The loss on a single example z is ℓ(z, θ). The empirical risk, which the model aims to minimize, is R(θ) = (1/n) Σ_i ℓ(z_i, θ) + λΩ(θ), where Ω(θ) is a regularization term with weight λ. The Hessian of the risk is the matrix of second derivatives, H = ∇_θ^2 R(θ). Our goal is to understand the influence of a training point z_i on a prediction for a test point z_test.

TDA methods are most cleanly applied in a supervised learning context, such as fine-tuning a language model on a specific task. The dataset is well-defined, accessible, and of a manageable size. Applying TDA to the pretraining stage of a foundation model is significantly harder due to the immense scale and frequent inaccessibility of the pretraining corpus.

Different TDA methods also require different levels of access to the model and training process:

• Some require full access to gradients and the ability to compute Hessian-vector products.
• Some rely on having saved model checkpoints from various stages of training.
• Others are only applicable to models with specific architectures, like a linear final layer.
• The most computationally demanding methods may require retraining the model on many different subsets of the data.

Contributive Methods (Core Families)

Contributive methods form the backbone of TDA. They estimate the positive (helpful) or negative (harmful) impact of each training point on a specific outcome.

(a) Influence Functions

Method
Influence Functions, introduced to machine learning by Koh & Liang (2017), are a classic technique from robust statistics. The core idea is to approximate the effect of removing a training point by asking: "How would the model's optimal parameters θ change if we infinitesimally up-weighted a single training point z_i?" This change in parameters is then used to measure the change in loss on a test point z_test.

Math
The effect on the parameters θ from up-weighting a training point z_i by a small amount ε is given by the influence function:

dθ_ε / dε |_(ε=0) = - (1/n) H⁻¹ ∇_θ ℓ(z_i, θ)

This equation tells us that the change in parameters is proportional to the gradient of the training point's loss, scaled by the inverse Hessian H⁻¹. The Hessian measures the curvature of the loss landscape; inverting it translates a step in gradient space into a change in parameter space.

To find the influence of z_i on the loss at z_test, we use the chain rule. The influence score, IF(z_i → z_test), is defined as the change in the test loss:

IF(z_i → z_test) = - (1/n) ∇_θ ℓ(z_test, θ)ᵀ H⁻¹ ∇_θ ℓ(z_i, θ)

A positive score means that up-weighting z_i increases the test loss, implying z_i is harmful to the prediction on z_test. A negative score means z_i is helpful.

Assumptions
Influence Functions rest on strong assumptions. They require the loss function to be twice-differentiable and strictly convex, ensuring a unique minimum and an invertible Hessian. The method is formally derived as a first-order approximation, meaning it is most accurate for small perturbations and assumes that the model parameters are already at a local minimum.

Computation
The primary computational bottleneck is calculating and inverting the p x p Hessian, H, which is infeasible for models with millions or billions of parameters. Instead, we never form H explicitly. We compute the term H⁻¹ v (where v is a vector, like ∇_θ ℓ(z_i, θ)) using iterative algorithms like the conjugate gradient method. These algorithms only require access to Hessian-vector products (HVPs), which can be computed efficiently with automatic differentiation. A damping term is often added to the Hessian diagonal to improve stability.

Strengths and Failure Modes

• Strengths: Influence Functions provide a fine-grained, theoretically grounded attribution score for each training point. They have been successfully used to identify mislabeled examples, understand dataset shortcuts, and debug model predictions.
• Failure Modes: The convexity assumption is strongly violated by deep neural networks. The local quadratic approximation that underpins Influence Functions may not hold in the complex, non-convex loss landscapes of Transformers. This can make the resulting scores fragile and unreliable, especially for out-of-distribution training points.

Caveat: Influence Function scores for deep, non-convex models should be treated with caution. They can be a powerful signal, but their fragility means they should ideally be validated with more direct checks, such as leave-one-out retraining on a small subset of the most influential points. Think of them as a sophisticated heuristic, not ground truth.

(b) TracIn

Method
TracIn (Pruthi et al., 2020) offers a more direct, empirical approach. Instead of approximating the effect of removal from a single final model, TracIn calculates influence by integrating the impact a training point has over the entire training trajectory. The intuition is simple: if, during training, the gradient for z_i frequently points in the same direction as the gradient for z_test, then the updates driven by z_i were likely helpful for z_test.

Math
The TracIn score is the sum of dot products between the gradients of the test point and the training point, evaluated at various checkpoints θ_k saved during training, and scaled by the learning rate η_k:

TracIn(z_i → z_test) ≈ Σ_(k∈K) η_k ∇_θ ℓ(z_test, θ_k) ⋅ ∇_θ ℓ(z_i, θ_k)

A large positive score indicates that z_i consistently pushed the model in a direction that would also reduce the loss on z_test, making it an influential proponent. A large negative score indicates it was an opponent.

Assumptions
TracIn makes fewer theoretical assumptions than Influence Functions. It does not require convexity. Its primary practical assumption is the availability of a representative set of model checkpoints K spanning the training process. The quality of the attribution depends directly on the quality and frequency of these checkpoints.

Computation
TracIn is computationally more straightforward than Influence Functions. It requires saving model checkpoints periodically during training. The attribution step involves a forward and backward pass for each training point and the test point at each checkpoint to compute the gradients. The cost scales linearly with the number of training points and the number of checkpoints.

Strengths and Failure Modes

• Strengths: Because it directly uses the training trajectory, TracIn is often more robust and reliable for deep, non-convex models than Influence Functions. It is conceptually simple and relatively easy to implement.
• Failure Modes: The scores are highly dependent on the choice of checkpoints. If only the final checkpoints are used, TracIn can miss important influences from early in training. The score is also sensitive to the learning rate schedule and optimizer dynamics (e.g., momentum).

Intuition: TracIn's power comes from its direct connection to what the optimizer actually did. It follows the path taken, while Influence Functions analyze the local geometry at the destination. For the winding roads of non-convex optimization, the path often tells a more accurate story than a local map of the final location.

(c) Representer Points

Method
The Representer Theorem provides a way to express the solution of certain kernel methods as a linear combination of the training examples. Representer Point Selection (Yeh et al., 2018) adapts this idea to deep learning. Under specific architectural assumptions, it allows us to decompose a network's pre-activation output for a test point into a weighted sum of similarities to the training points. The weights, or "representer values," serve as the attribution scores.

Math
For a model with a final linear layer that minimizes an L2-regularized objective, the prediction function f(x) can be approximated as a linear combination of kernel similarities between the test input's representation φ(x) and each training input's representation φ(x_i):

f(x) ≈ Σ_i α_i φ(x_i)ᵀ φ(x)

The α_i are the representer values. A positive α_i means that z_i is an "excitatory" example, pushing the prediction in a positive direction. A negative α_i means it is "inhibitory." These values can be computed efficiently from the model's final-layer parameters and the loss gradients.

Assumptions
This method's applicability is constrained by its strong architectural assumptions: the model must be trained with L2 regularization on its final layer parameters, and the final output must be a linear function of the learned representations φ(x). While this holds for many standard classification networks, it may not apply directly to more complex architectures like Transformers without modification.

Computation
Once the model is trained, computing the representer values is typically fast. It involves a single backward pass to get gradients and then solving a system of linear equations that depends on the training set size. The main cost is in pre-computing and storing the representations φ(x_i) for all training data.

Strengths and Failure Modes

• Strengths: The method provides directly interpretable signed attributions (excitatory vs. inhibitory), which is a powerful debugging tool. When its assumptions are met, it is computationally efficient.
• Failure Modes: Its primary limitation is the strict set of architectural assumptions. It is not a general-purpose method for arbitrary deep learning models. The quality of the attribution also depends on the quality of the learned representations φ(x).

(d) Data-Shapley

Method
Data-Shapley (Ghorbani & Zou, 2019; Jia et al., 2019) brings a game-theoretic approach to TDA. It treats the training data points as players in a cooperative game where the goal is to achieve high model performance. The Shapley value is a concept from game theory that provides a principled way to fairly distribute the total "payout" (model utility) among the players. The Shapley value of a data point is its average marginal contribution to the model's performance across all possible subsets (coalitions) of the training data.

Math
Let U(S) be a utility function that measures the performance of a model trained on a subset of data S ⊆ D. The Shapley value ϕ_i for a data point z_i is:

ϕ_i = E_π [U(S_π ∪ {i}) - U(S_π)]

Here, the expectation is taken over all random permutations π of the training data, and S_π is the set of all data points preceding i in that permutation. In words, it's the average improvement in performance gained by adding z_i to a random subset of data.

Assumptions
Data-Shapley's main assumption is the choice of a meaningful utility function U. This could be validation accuracy, log-likelihood, or some other performance metric. The "fairness" of the attribution is only as good as the utility function is representative of the desired model behavior.

Computation
Computing exact Shapley values is computationally prohibitive, as it requires training the model on all 2^n subsets of the data. In practice, approximations are used. The most common is the Truncated Monte Carlo (TMC) estimation, which samples random permutations of the data and stops adding points once the marginal contributions stabilize. For specific models like k-Nearest Neighbors, efficient closed-form or near-closed-form variations like KNN-Shapley exist.

Strengths and Failure Modes

• Strengths: Data-Shapley is founded on a solid, axiomatic basis (efficiency, symmetry, linearity, null player), making it a "fair" measure of contribution. It is particularly well-suited for dataset-level tasks like data valuation for monetization or identifying low-value points for dataset pruning.
• Failure Modes: The computational cost is its greatest weakness, making it impractical for large models and datasets without significant approximations. The results can have high variance due to the stochasticity of both model training and the sampling process. Furthermore, the choice of utility function can drastically change the resulting attributions.

Opinion: Data-Shapley is the gold standard in theory but often a computational nightmare in practice. It is best reserved for situations where you need to make high-stakes decisions about dataset composition (e.g., pruning, valuation) and have the resources to invest in robust approximation. For routine prediction-level debugging, simpler methods are often more practical.

Bridge to LLMs

Applying these methods directly to large language model (LLM) pretraining is fraught with challenges. The sheer scale of the data, its often-proprietary nature, and the non-stationary training dynamics make full attribution analysis infeasible.

However, TDA is highly actionable and relevant for the fine-tuning stages of LLMs (e.g., supervised fine-tuning (SFT) or instruction tuning). In this regime, the datasets are curated, accessible, and orders of magnitude smaller. Here, TDA can help us understand:

• Which instruction-tuning examples are most responsible for a specific alignment behavior?
• Which documents in a domain-adaptation dataset are causing catastrophic forgetting of general abilities?
• Why does the model fail on a specific query after being fine-tuned on our data?

The findings of Razeghi et al. (2022) provide the perfect motivation. We know that corpus-level statistics shape a model's core capabilities. Fine-tuning is our chance to precisely perturb those capabilities. TDA is the microscope that lets us see which examples in our fine-tuning data are driving those changes, for better or for worse.

Synthesis & Guidance

With four families of methods, how should a practitioner choose?

• For quick, scalable debugging of specific predictions on deep models: Start with TracIn. It requires saving checkpoints but is empirically robust and sidesteps the theoretical pitfalls of Influence Functions in non-convex settings.
• For finding mislabeled data or when a local approximation is sufficient: Use Influence Functions, but be prepared to validate the results. The HVP-based implementations scale well.
• If your model has a linear readout and is trained with L2 regularization: Representer Points offer uniquely interpretable, signed attributions that can distinguish helpful from harmful examples.
• For dataset-level valuation, pruning, or making policy decisions about data subsets: Use Data-Shapley approximations. This is the most principled but also the most expensive option.

A robust workflow often involves triangulation. If TracIn and Influence Functions both flag the same training example as highly influential for a given error, your confidence in that attribution should be much higher. Before taking a drastic action like removing data, seek consensus across multiple methods. When reporting results, be transparent about your assumptions: which checkpoints were used for TracIn, what damping was used for Influence Functions, or which utility function was chosen for Shapley.

Limits & Open Questions

Training Data Attribution is an active and evolving field of research. Major open questions remain:

• Scalability: How can we develop TDA methods that are truly feasible for foundation model pretraining corpora?
• Memorization vs. Contribution: What is the precise relationship between an example being influential and being memorized? An example can teach a generalizable concept without being memorized, and a memorized example might have little influence on other predictions.
• Connection to Unlearning: Can we leverage TDA to perform more efficient and effective "machine unlearning," where the goal is to surgically remove a model's knowledge of specific data points for privacy or safety reasons?
• Complex Objectives: How do we perform attribution in more complex training pipelines like Reinforcement Learning from Human Feedback (RLHF), where the "loss function" is itself a learned reward model?

TDA is more than an academic exercise; it is a fundamental tool for building more reliable, transparent, and debuggable machine learning systems. By connecting model behavior back to the data it learned from, we take a critical step from opaque prediction engines to understandable and trustworthy tools.

References

• Ghorbani, A., & Zou, J. (2019). "Data-Shapley: Equitable Valuation of Data for Machine Learning". Proceedings of the 36th International Conference on Machine Learning (ICML).
• Jia, R., et al. (2019). "Towards Efficient Data Valuation Based on the Shapley Value". Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS).
• Koh, P. W., & Liang, P. (2017). "Understanding Black-box Predictions via Influence Functions". Proceedings of the 34th International Conference on Machine Learning (ICML).
• Pruthi, G., et al. (2020). "TracIn: An Influence-based Approach for Tracing Training Data". Advances in Neural Information Processing Systems (NeurIPS).
• Razeghi, Y., et al. (2022). "Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning". Findings of the Association for Computational Linguistics: EMNLP 2022.
• Tenney, I. (2023). "A Hitchhiker's Guide to Training Data Attribution".
• Yeh, C.-K., et al. (2018). "Representer Point Selection for Explaining Deep Neural Networks". Advances in Neural Information Processing Systems (NeurIPS).