When VLMs Need a Human Safety Net: Building Document Extraction with Confidence Calibration

Healthcare document processing has a fundamental constraint: you can't be wrong. A misread medication dosage or transposed patient ID isn't a minor error—it's a potential safety incident. Yet vision-language models, for all their power, are confidently wrong with uncomfortable frequency.

Inspired by Tennr's approach to healthcare automation, I built a document extraction platform that embraces this reality. The core insight: the model's job isn't just to extract data—it's to know when it shouldn't be trusted. When confidence is low, the system routes to human review. When humans correct errors, those corrections feed back into model improvement.

Here's what I learned building it.

The Problem: Models Don't Know What They Don't Know

Vision-language models like olmOCR can extract structured data from complex medical forms with impressive accuracy. But they have a fatal flaw: their confidence scores are poorly calibrated. A model that outputs 90% confidence should be right 90% of the time—but in practice, these models are often overconfident on novel inputs and underconfident on routine ones.

This creates two failure modes:

The solution isn't better models. It's calibration: learning to transform raw model outputs into reliable confidence scores, and using those scores to make intelligent routing decisions.

The Architecture: Confidence-Driven Routing

┌─────────────────────────────────────────────────────────────────────────────┐
│                          DOCUMENT INGESTION                                  │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐               │
│  │ PDF/Image    │      │ Preprocessing│      │ Layout       │               │
│  │ Upload       │ ───▶ │ Enhancement  │ ───▶ │ Detection    │               │
│  │ (API/CLI)    │      │ (denoise,    │      │ (regions,    │               │
│  │              │      │  deskew)     │      │  tables)     │               │
│  └──────────────┘      └──────────────┘      └──────┬───────┘               │
└─────────────────────────────────────────────────────┼───────────────────────┘
                                                      │ regions
                                                      ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          VLM INFERENCE LAYER                                 │
│                                                                              │
│  ┌────────────────────┐      ┌────────────────────────────────────────┐     │
│  │  olmOCR-2-7B-MLX   │      │  Self-Consistency Engine               │     │
│  │  ─────────────     │ ───▶ │  ─────────────────────                 │     │
│  │  Quantized VLM     │      │  k=1 (text) / k=3 (mixed) / k=5 (hard) │     │
│  │  MLX acceleration  │      │  Majority voting + agreement ratio     │     │
│  └────────────────────┘      └─────────────────┬──────────────────────┘     │
└────────────────────────────────────────────────┼────────────────────────────┘
                                                 │ predictions + raw confidence
                                                 ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          CALIBRATION LAYER                                   │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────┐     │
│  │  Temperature Scaling Calibrator                                     │     │
│  │                                                                     │     │
│  │  raw_confidence ───▶ logit ───▶ scale by T ───▶ calibrated_conf    │     │
│  │                                                                     │     │
│  │  Per-field calibration: T_medication=1.5, T_name=0.9, T_date=0.8  │     │
│  └────────────────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────┬───────────────────────────┘
                                                  │ calibrated confidence
                                                  ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          ROUTING DECISION                                    │
│                                                                              │
│                    ┌─────────────────────────────┐                          │
│                    │  Confidence ≥ threshold?    │                          │
│                    │  (per-field: 85%-98%)       │                          │
│                    └──────────┬──────────────────┘                          │
│                               │                                              │
│              ┌────────────────┼────────────────┐                            │
│              │                │                │                            │
│              ▼ YES            ▼ MAYBE          ▼ NO                         │
│  ┌───────────────────┐ ┌─────────────────┐ ┌───────────────────┐            │
│  │ AUTOPILOT         │ │ FLAG FOR REVIEW │ │ HUMAN REVIEW      │            │
│  │ ───────────       │ │ ───────────     │ │ ────────────      │            │
│  │ Direct export     │ │ Auto-assign but │ │ Queue for manual  │            │
│  │ to EHR/database   │ │ mark for verify │ │ annotation        │            │
│  └───────────────────┘ └─────────────────┘ └─────────┬─────────┘            │
└─────────────────────────────────────────────────────────┼───────────────────┘
                                                          │
                                                          ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          HUMAN-IN-THE-LOOP                                   │
│                                                                              │
│  ┌────────────────────┐      ┌────────────────────────────────────────┐     │
│  │  Streamlit UI      │      │  Active Learning Selector              │     │
│  │  ─────────────     │      │  ────────────────────                  │     │
│  │  Priority queue    │ ◀──▶ │  Uncertainty × 0.5 +                   │     │
│  │  Keyboard shortcuts│      │  Diversity × 0.3 +                     │     │
│  │  Context display   │      │  Value × 0.2                           │     │
│  │  Audit trail       │      │                                        │     │
│  └─────────┬──────────┘      └────────────────────────────────────────┘     │
│            │ corrections                                                     │
└────────────┼────────────────────────────────────────────────────────────────┘
             │
             ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                          FEEDBACK LOOP                                       │
│                                                                              │
│  ┌────────────────────┐      ┌────────────────────┐                         │
│  │  Dataset Manager   │      │  Calibration       │                         │
│  │  ─────────────     │      │  Monitor           │                         │
│  │  Version annotations│ ◀──▶│  Refit temperature │                         │
│  │  Trigger retraining│      │  every N samples   │                         │
│  └────────────────────┘      └────────────────────┘                         │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────┐     │
│  │  Observability: MLflow (experiments) + Prometheus + Grafana        │     │
│  └────────────────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────────────────┘

Documents flow through preprocessing (image enhancement, layout detection) into the VLM inference layer, where olmOCR-2-7B-MLX extracts structured fields. The self-consistency engine runs multiple inferences for difficult regions, using agreement ratio as a confidence signal.

The calibration layer transforms raw model outputs into reliable probabilities using temperature scaling—learned separately per field type from human correction data. The routing layer then decides: autopilot (high confidence), flag-for-review (borderline), or human review (low confidence).

The human-in-the-loop layer provides a Streamlit-based review interface with priority queuing and keyboard shortcuts. The active learning selector chooses which samples are most valuable to annotate.

Finally, the feedback loop closes the circle: corrections update the calibrator, trigger dataset versioning, and feed into model retraining. The entire system is instrumented with MLflow for experiments and Prometheus/Grafana for operational metrics.

The target: autopilot handles 70%+ of documents without human intervention, while maintaining accuracy guarantees on the remaining 30% through targeted review.

Lesson 1: Self-Consistency Beats Single Inference

My first approach used a single inference pass: run the VLM once, take the output, use the logit-based confidence. The problem? Single-pass confidence is noisy. The same input might produce different outputs on different runs, but a single inference can't reveal that instability.

The fix is self-consistency sampling: run inference k times with different random seeds, then use majority voting to select the consensus answer. The agreement ratio becomes a confidence signal—if 5 out of 5 samples agree, you have high confidence; if it's 3-2, something is ambiguous.

async def run_self_consistency(image, prompt, adapter, k=3):
    # Run k parallel inferences with different seeds
    predictions = await asyncio.gather(*[
        run_inference(image, prompt, seed=i) for i in range(k)
    ])

    # Majority voting: serialize predictions, count occurrences
    serialized = [json.dumps(p, sort_keys=True) for p in predictions]
    most_common, count = Counter(serialized).most_common(1)[0]

    agreement_ratio = count / k
    return json.loads(most_common), agreement_ratio

The intuition: if the model is uncertain, different seeds will produce different outputs. If the model is confident and correct, all samples will converge. Self-consistency surfaces the former while confirming the latter.

There's a cost tradeoff: k=3 means 3x inference time. For simple text fields, k=1 is fine. For complex regions (handwriting, tables, checkboxes), k=5 is worth the latency. The system uses region type to dynamically select k.

The lesson: Single-pass confidence is unreliable. Self-consistency sampling reveals model uncertainty that logits alone hide.

Lesson 2: Raw Probabilities Aren't Probabilities

Even with self-consistency, raw confidence scores are miscalibrated. A model might report 85% confidence and be right 95% of the time—or report 85% and be right only 70% of the time. Without knowing which, you can't set reliable routing thresholds.

Temperature scaling solves this. You collect a validation set of (confidence, is_correct) pairs, then learn a single temperature parameter T that rescales logits:

class TemperatureScaling:
    def fit(self, confidences, correctness):
        # Convert confidences to logits
        logits = np.log(confidences / (1 - confidences))

        # Optimize T to minimize negative log-likelihood
        def nll(T):
            scaled = 1 / (1 + np.exp(-logits / T))
            return -np.mean(correctness * np.log(scaled) +
                           (1 - correctness) * np.log(1 - scaled))

        result = minimize(nll, x0=1.0, bounds=[(0.1, 10.0)])
        self.temperature = result.x[0]

    def calibrate(self, confidence):
        logit = np.log(confidence / (1 - confidence))
        return 1 / (1 + np.exp(-logit / self.temperature))

After calibration, if the model says 95%, it's right 95% of the time (±calibration error). This makes threshold-based routing meaningful: "send to review if confidence < 95%" actually means something.

The calibrator retrains continuously. Every human correction is a calibration sample. A CalibrationMonitor accumulates samples and refits temperature every N predictions, tracking Expected Calibration Error (ECE) over time.

The lesson: Raw model confidence is a ranking signal, not a probability. Temperature scaling transforms it into something you can actually trust for decision-making.

Lesson 3: Human Corrections Are Your Most Valuable Data

Every document routed to human review is an opportunity. When a reviewer corrects a field, you get ground truth on a case the model found difficult. That's exactly the data you need to improve.

But not all corrections are equally valuable. A typo fix on an easy case teaches less than a structural correction on a hard one. Active learning selects which samples to prioritize for annotation based on their expected learning impact.

The selection strategy combines three signals:

def select_annotation_batch(predictions):
    # Uncertainty: low-confidence samples are more informative
    uncertainty_scores = 1 - np.array([p['confidence'] for p in predictions])

    # Diversity: cluster samples, prefer underrepresented areas
    diversity_scores = compute_cluster_distances(predictions)

    # Value: high-difficulty, common patterns have more impact
    value_scores = predictions['difficulty'] * 0.6 +
                   predictions['frequency'] * 0.4

    # Weighted combination
    total = (0.5 * uncertainty_scores +
             0.3 * diversity_scores +
             0.2 * value_scores)

    return top_k_indices(total, k=batch_size)

Uncertainty sampling alone leads to annotating outliers that don't generalize. Diversity sampling ensures coverage. Value scoring weights toward patterns that appear frequently—fixing a common error type helps more than fixing a rare one.

The feedback loop: corrections flow into a versioned annotation store, triggering dataset updates and model retraining on a configurable schedule. Each retrained model gets validated against a golden test set before deployment.

The lesson: Human review isn't just a safety net—it's a data generation mechanism. Active learning extracts maximum value from every correction.

Lesson 4: Different Fields Need Different Confidence

A name field and a medication dosage field have different error costs. Getting the name slightly wrong (Robert vs Roberto) is recoverable. Getting the dosage wrong (15 mg vs 150 mg) could cause patient harm.

Rather than a single confidence threshold, the system uses per-field calibration. High-stakes fields like medication names and dosages have strict thresholds (98%). Lower-stakes fields like clinic names have relaxed thresholds (90%).

Field types also get different self-consistency settings. Checkboxes use k=5 because they're error-prone. Printed text uses k=1 because it's usually unambiguous. The prompt selector chooses templates based on region type, and the confidence scorer combines region difficulty with field-specific calibration.

thresholds = {
    'medication_name': 0.98,
    'dosage': 0.98,
    'patient_name': 0.95,
    'date': 0.95,
    'clinic_name': 0.90,
    'general_text': 0.85,
}

if confidence < thresholds[field_type]:
    route_to_human_review(document, field)

The calibrator maintains separate temperature parameters per field type. Medication fields might need T=1.5 (model is overconfident), while dates might need T=0.8 (model is underconfident). The system learns these automatically from correction data.

The lesson: Not all fields are equal. Calibrate and threshold based on error cost, not just raw accuracy.

Lesson 5: The Annotation Interface Is a Product

I initially treated the human review UI as an afterthought—a simple form to view extractions and submit corrections. That was a mistake. Reviewer efficiency directly impacts system economics. A UI that takes 60 seconds per correction costs twice as much as one that takes 30 seconds.

The Streamlit-based review interface became a first-class product:

• Priority queue: Items sorted by confidence (lowest first), field type (high-stakes first), and document age (oldest first)
• Context preservation: Show the original image region alongside the extracted text, with bounding boxes highlighted
• Keyboard shortcuts: Accept, reject, and edit without touching the mouse
• Audit trail: Every correction tracked with reviewer ID, timestamp, and before/after values
• Bulk actions: Accept all high-confidence items with one click when the queue backs up

The queue also shows aggregate metrics: current backlog size, average review time, acceptance rate by field type. Reviewers can see patterns—if checkbox extractions are consistently wrong, that's feedback to the model team.

The lesson: Human-in-the-loop is only sustainable if the loop is fast. Invest in annotation UX as if it were user-facing (it is).

Lesson 6: Observability Is Non-Negotiable

In a system with this many moving parts—preprocessing, inference, calibration, routing, review, retraining—things break in subtle ways. A calibrator that stops updating. A queue that grows faster than reviewers can drain it. A retrained model that's worse than its predecessor.

The observability stack tracks everything:

• MLflow: Every inference logged with input hash, prompt version, output, confidence, latency. Every model version registered with metrics and lineage.
• Prometheus: Gauges for queue depth, p50/p95 inference latency, autopilot vs review ratio, calibration error.
• Grafana dashboards: Real-time visualization of throughput, accuracy by field type, reviewer productivity, model comparison.

Every request carries a correlation ID that propagates through all services. When something goes wrong, you can trace from API request → preprocessing → inference → calibration → routing → review → export in a single query.

Alerts fire on SLO breaches: queue depth > 1000, p95 latency > 30s, ECE > 0.05. The on-call has dashboards and runbooks to diagnose and recover.

The lesson: ML systems fail silently. Instrument everything, alert on meaningful thresholds, and make debugging possible before you need it.

What I'd Do Differently

1. Start with calibration, not model architecture: I spent too long optimizing the VLM before realizing that calibration was the bottleneck. A well-calibrated weaker model beats an uncalibrated stronger one for production routing decisions.

2. Build synthetic data generation earlier: The system includes a synthetic form generator for testing, but it came late. Having diverse synthetic data from day one would have caught more edge cases before production.

3. Add fallback OCR sooner: For text-heavy regions, traditional OCR (Tesseract, PaddleOCR) is often faster and just as accurate. The system now routes by region type, but initially it VLM'd everything.

4. Implement canary deployments for prompts: Prompts are code. Changes can break extraction in subtle ways. The system now versions prompts and supports canary rollouts, but we learned this after a prompt change caused a 15% accuracy regression.

Key Takeaways

This project is a medical document processing platform built with olmOCR-2-7B-MLX, FastAPI, MLflow, and Streamlit. Designed for HIPAA-compliant healthcare workflows with human-in-the-loop quality assurance.