Aegis-Building a Fraud Detection System

Insurance fraud costs the industry over $80 billion annually. Yet most fraud detection systems still rely on simple rule-based approaches: flag claims over $X, or those from first-time claimants. The problem? Fraudsters adapt, and rules don't.

I built Aegis, a fraud detection system that combines the semantic understanding of language models with the interpretability of gradient boosting. The result: 75% precision at 50% recall—meaning we catch half of all fraud while being right 3 out of 4 times. That's a 12x improvement over the manual baseline.

Here's what I learned building it.

The Problem: Fraud Hides in Plain Sight

Insurance claims have an inherent asymmetry: only ~6% are fraudulent. This means any system that simply predicts "not fraud" for everything achieves 94% accuracy—while being completely useless.

The real challenge isn't classification accuracy. It's prioritization: which claims should investigators spend their limited time on?

Manual review has two failure modes:

I needed a system that could surface high-risk claims without drowning investigators in false alarms.

The Architecture: Two Models Are Better Than One

The key insight came from analyzing what makes fraud claims different. Fraudsters leave two kinds of traces:

No single model handles both well. Transformers excel at understanding language but struggle with tabular data. Gradient boosting crushes tabular features but can't parse free text.

So I built a dual-engine system:

┌─────────────────────────────────────────────────────────────┐
│                     Claim Submission                         │
│  ┌──────────────────┬────────────────────────────────────┐  │
│  │ Structured Data  │     Accident Narrative             │  │
│  │ • Age: 26-39     │  "I was stopped at a light when   │  │
│  │ • Past accidents │   another car sort of hit me..."  │  │
│  │ • DUI history    │                                    │  │
│  └────────┬─────────┴──────────────────┬─────────────────┘  │
│           │                             │                    │
│           ▼                             ▼                    │
│  ┌─────────────────┐       ┌─────────────────────────────┐  │
│  │    XGBoost      │◄──────│   DistilRoBERTa + LoRA      │  │
│  │ (Decision Core) │       │   (Fraud Signal Extractor)  │  │
│  └────────┬────────┘       └─────────────────────────────┘  │
│           │                                                  │
│           ▼                                                  │
│  ┌─────────────────┐                                        │
│  │   Calibrator    │                                        │
│  │  (Isotonic Reg) │                                        │
│  └────────┬────────┘                                        │
│           │                                                  │
│           ▼                                                  │
│     Risk Score: 73/100                                       │
│     Level: HIGH                                              │
│     Top Factor: "Vague Language"                             │
└─────────────────────────────────────────────────────────────┘

The language model extracts three fraud signals from the narrative:

• Vague language ("sort of happened", "hard to say")
• Timeline inconsistencies (conflicting dates, unclear sequence)
• Pre-existing damage mentions (suspicious references to prior wear)

These become features fed into XGBoost alongside traditional structured data.

Lesson 1: LoRA Makes Fine-Tuning Practical

The standard approach to fine-tuning a transformer is updating all 82 million parameters. That's expensive, slow, and produces massive model files.

LoRA (Low-Rank Adaptation) changes the game. Instead of updating the full weight matrices, it learns low-rank decomposition matrices that get added to specific layers—typically just the attention weights.

# The magic of LoRA: only 1.5% of parameters are trainable
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,                    # Rank of decomposition
    lora_alpha=16,          # Scaling factor
    target_modules=["query", "value"],  # Only attention layers
    lora_dropout=0.1,
    task_type="SEQ_CLS"
)

model = get_peft_model(base_model, lora_config)
# trainable params: 1.2M (1.5% of 82M)
# model size: 30MB instead of 500MB

The result? Training time dropped 70%, and the model file went from 500MB to 30MB—while retaining 97% of full fine-tuning performance.

The lesson: Parameter-efficient fine-tuning isn't just for saving money. Small models deploy faster, cache better, and run on cheaper hardware.

Lesson 2: Multi-Label Beats Binary

My first approach was to train a binary classifier: "Is this claim fraudulent?"

It didn't work. The model learned to output ~6% probability for everything—matching the base rate while being useless for prioritization.

The fix was reframing the problem. Instead of predicting fraud directly, I trained the LLM to detect specific fraud signals:

# Multi-label classification: one forward pass, three signals
def get_llm_features(text, model, tokenizer):
    inputs = tokenizer(text, return_tensors="pt", max_length=128)

    with torch.no_grad():
        outputs = model(**inputs)
        probs = torch.sigmoid(outputs.logits).cpu().numpy()[0]

    return {
        'has_vague_language_score': float(probs[0]),
        'mentions_preexisting_damage_score': float(probs[1]),
        'has_timeline_inconsistency_score': float(probs[2])
    }

This approach has three advantages:

The lesson: When direct prediction fails, reframe the task to predict intermediate signals instead. Your downstream model can learn the complex mapping from signals to outcomes.

Lesson 3: Monotonic Constraints Encode Domain Knowledge

Here's a problem I didn't anticipate: the model learned that more driving experience increased fraud risk.

Wait, what?

It turned out the training data had a spurious correlation: older, experienced drivers filed higher-value claims (they drove nicer cars). The model picked this up as a fraud signal.

XGBoost has an elegant solution: monotonic constraints. You can tell the model that certain features must have one-directional effects:

# Constraints: +1 = must increase risk, -1 = must decrease, 0 = unconstrained
monotone_constraints = {
    'SPEEDING_VIOLATIONS': 1,    # More violations → higher risk
    'DUIS': 1,                   # More DUIs → higher risk
    'PAST_ACCIDENTS': 1,         # More accidents → higher risk
    'DRIVING_EXPERIENCE': -1,    # More experience → lower risk
    'has_vague_language_score': 1,
    'mentions_preexisting_damage_score': 1,
    'has_timeline_inconsistency_score': 1
}

xgb_model = XGBClassifier(
    monotone_constraints=tuple(monotone_constraints.values()),
    # ... other params
)

During tree construction, XGBoost only considers splits that satisfy these constraints. No amount of spurious correlation can override them.

The lesson: Encode what you know. Monotonic constraints aren't just for interpretability—they regularize against data artifacts and make predictions defensible to regulators.

Lesson 4: Calibration Is Non-Negotiable

Here's a common mistake: treating model probabilities as real probabilities.

When my XGBoost model output 0.8, did that mean an 80% chance of fraud? Absolutely not. Raw model scores are uncalibrated—they measure relative ranking, not actual probabilities.

This matters because:

• Investigators need to set thresholds ("review all claims above 50%")
• Business decisions require expected value calculations
• "90% fraud probability" must mean 9 in 10 similar claims are actually fraudulent

I used isotonic regression for calibration:

from sklearn.isotonic import IsotonicRegression

# Fit on validation set
calibrator = IsotonicRegression(out_of_bounds='clip')
calibrator.fit(val_proba_uncalibrated, val_labels)

# At inference time
raw_probability = xgb_model.predict_proba(X)[:, 1]
calibrated_probability = calibrator.predict(raw_probability)
risk_score = int(calibrated_probability * 100)

Isotonic regression learns a non-decreasing mapping from raw scores to true probabilities. Unlike Platt scaling (which assumes logistic form), it makes no parametric assumptions—crucial when your model outputs have weird distributions.

The lesson: Raw probabilities are not probabilities. Always calibrate before presenting scores to users or making threshold-based decisions.

Lesson 5: Speed and Size Matter More Than You Think

My first version used SHAP for per-prediction explanations. Each prediction took 30 seconds. That's fine for batch processing, but useless for a real-time dashboard.

I made a pragmatic trade-off: global feature importance instead of per-prediction explanations.

# Fast: global feature importance (~0ms)
feature_importance = xgb_model.feature_importances_
top_indices = np.argsort(feature_importance)[-3:][::-1]

# Slow: per-prediction SHAP (~30,000ms)
# explainer = shap.TreeExplainer(xgb_model)
# shap_values = explainer.shap_values(X)

The full optimization stack:

• LoRA adapters: 500MB → 30MB
• Joblib serialization: Efficient model caching
• Streamlit @cache_resource: Models load once, persist across requests
• Feature importance over SHAP: 30s → 300ms

Final footprint: 7MB total model artifacts, ~320ms inference time.

The lesson: Optimize for deployment from the start. A model that can't run in production is just a Jupyter notebook.

What I'd Do Differently

1. Use more sophisticated text augmentation: I generated synthetic narratives with rules. LLM-based augmentation would create more realistic training data.

2. Add image features: Real fraud detection uses damage photos. A future version could incorporate vision models for inconsistency detection.

3. Implement drift detection: Fraud tactics evolve. The model needs monitoring for distribution shift and scheduled retraining triggers.

4. A/B test thresholds: The 70/40 thresholds for HIGH/MEDIUM risk were chosen heuristically. In production, you'd optimize these against business metrics.

The Results

The 75% precision means: if investigators review our top-ranked claims until they've found half of all fraud, they'll be right 3 out of 4 times. Compared to random selection (~6% fraud rate), that's a 12x improvement in investigator efficiency.

Key Takeaways

Aegis is open source and available on GitHub. The training pipeline, including LoRA fine-tuning and XGBoost calibration, runs on Colab.