Voice agents break in ways you don't expect. A prompt that handles "I need an oil change" perfectly in testing fails when a customer says "my car's making a weird noise, probably needs service." The intent is the same, but the phrasing is different enough to confuse the classifier.
This is manageable when you're running one agent for one location. You see the failure in logs, you tweak the prompt, you move on. But when the same agent runs across 50 dealerships, each generating hundreds of calls per day with their own regional phrasings and edge cases, manual monitoring becomes impossible. You can't have an engineer watching dashboards all day, and you certainly can't have one responding to 3 AM failures.
I built a system that closes this loop automatically. It detects when the agent starts failing, identifies what's going wrong, generates a better prompt, and deploys it—all without waking anyone up. The key insight is treating prompt degradation the same way we treat infrastructure failures: as something that can be detected via metrics and remediated via automation.
The Problem with Voice AI at Scale
The challenge isn't making a voice agent that works. The challenge is making one that keeps working as the world changes around it.
Consider what happens after you deploy a successful appointment-booking agent. Customers start calling with requests you didn't anticipate: "Can I get in tomorrow instead of Tuesday?" (a reschedule, not a new booking), "Do you guys do tire rotations?" (a service inquiry, not an appointment), "My husband's car, not mine" (a vehicle mismatch your slot-booking logic doesn't handle).
Each of these is a small failure. Individually, they're noise. But when they accumulate—when 15% of calls are failing instead of 5%—you have a systemic problem that requires prompt changes.
The traditional response is reactive. Someone notices the metrics are down. They pull call logs. They identify the failure pattern. They draft a new prompt. They test it locally. They deploy it. This cycle takes hours at best, days at worst. And during that time, customers are having bad experiences.
Automation can compress this cycle to minutes. The system watches its own metrics, detects when something breaks, diagnoses what's failing, generates a fix, and deploys it. The humans review the results in the morning, but the fires get put out overnight.
System Architecture
The solution requires three components working together:
The Voice Agent handles customer calls. It classifies intent, executes the appropriate conversation flow (booking, rescheduling, FAQ), and logs outcomes. Critically, it logs structured outcomes—not just "success" or "failure" but why it failed.
Keep is an open-source AIOps platform. It ingests metrics from the voice agent, applies anomaly detection (e.g., "conversion rate dropped 20% in the last hour"), and triggers remediation workflows. Keep handles the detection side: knowing something is wrong.
The GEPA Optimizer handles remediation. It receives failed call data from Keep, identifies patterns in the failures, generates an improved prompt, and deploys it as a new version. GEPA handles the fix side: doing something about the problem.
The three form a closed loop. Metrics flow from the agent to Keep. When thresholds are breached, Keep triggers the optimizer. The optimizer generates a new prompt and deploys it back to the agent. The loop runs continuously without human intervention.
┌─────────────────────────────────────────────────────────────────────────────┐
│ VOICE AGENT LAYER │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────┐ │
│ │ Intent Classifier │ │ Conversation │ │ Outcome Logger │ │
│ │ ───────────── │ ───▶ │ Flows │ ───▶ │ ─────────── │ │
│ │ • Booking │ │ • BookingFlow │ │ • success │ │
│ │ • Reschedule │ │ • RescheduleFlow │ │ • failure_type │ │
│ │ • FAQ │ │ • FAQFlow │ │ • transcript │ │
│ └────────────────────┘ └────────────────────┘ └───────┬────────┘ │
└──────────────────────────────────────────────────────────────────┼───────────┘
│ metrics
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ AIOPS LAYER (Keep) │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────┐ │
│ │ Metric Ingestion │ │ Anomaly Detection │ │ Workflow │ │
│ │ ───────────── │ ───▶ │ ───────────── │ ───▶ │ Trigger │ │
│ │ • conversion_rate │ │ "Rate dropped │ │ • fetch fails │ │
│ │ • avg_call_time │ │ 20% in 1 hour" │ │ • call webhook │ │
│ └────────────────────┘ └────────────────────┘ └───────┬────────┘ │
└──────────────────────────────────────────────────────────────────┼───────────┘
│ alert + data
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ OPTIMIZATION LAYER (GEPA) │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────┐ │
│ │ Failure Analysis │ │ Prompt Generation │ │ Version Store │ │
│ │ ───────────── │ ───▶ │ ───────────── │ ───▶ │ ─────────── │ │
│ │ NO_SLOTS: 18 │ │ Current + fails │ │ v1 → v2 → v3 │ │
│ │ CONFIDENCE: 8 │ │ + objectives │ │ │ │
│ │ → derive goals │ │ → LLM → v4 │ │ is_active: v4 │ │
│ └────────────────────┘ └────────────────────┘ └───────┬────────┘ │
└──────────────────────────────────────────────────────────────────┼───────────┘
│
┌────────────────────── deploys v4 ◀─────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ CLOSED LOOP: Agent uses new prompt → metrics recover → Keep auto-resolves │
└─────────────────────────────────────────────────────────────────────────────┘The architecture separates concerns cleanly. The agent focuses on handling calls well. Keep focuses on detecting problems. The optimizer focuses on generating fixes. Each component is independently testable and replaceable.
Structured Failure Logging
The entire system depends on one thing: knowing why calls fail, not just that they fail.
A log entry that says "call failed" tells you nothing actionable. Did it fail because no appointment slots were available? Because the customer hung up? Because the agent couldn't understand the request? Each of these requires a different fix. Slot availability is a data problem, not a prompt problem. Customer hangups might indicate the greeting is too long. Intent confusion indicates the prompt needs better examples.
I defined a canonical set of failure reasons:
class FailureReason(str, Enum):
NO_SLOTS = "no_slots"
CUSTOMER_DISENGAGED = "customer_disengaged"
AGENT_CONFIDENCE_LOW = "agent_confidence_low"
UNKNOWN = "unknown"Every call outcome includes the appropriate failure reason. When a customer hangs up mid-conversation, that's CUSTOMER_DISENGAGED. When the booking system has no available slots, that's NO_SLOTS. When the agent can't classify what the customer wants, that's AGENT_CONFIDENCE_LOW.
This structure propagates through the entire system. When the optimizer receives a batch of failed calls, it can aggregate by failure type: "Of these 30 failures, 18 were NO_SLOTS, 8 were AGENT_CONFIDENCE_LOW, and 4 were CUSTOMER_DISENGAGED." That breakdown tells you what to fix. If most failures are slot-related, the prompt probably needs to offer waitlist options more proactively. If most are confidence-related, the prompt needs better intent-handling instructions.
The key insight is that failure categorization is a design decision made at the agent level, but it pays dividends at the optimization level. You can't derive this structure from raw transcripts—the agent has to record it explicitly.
Deriving Optimization Objectives
Raw failure counts aren't useful to an LLM generating prompt improvements. "18 NO_SLOTS failures" doesn't tell the model what to do. The system needs to translate failure patterns into natural language objectives that guide generation.
This translation encodes domain knowledge about what each failure type implies:
def derive_objectives(failure_reasons: list[FailureReason]) -> list[str]:
counts = Counter(failure_reasons)
objectives = []
if counts.get(FailureReason.NO_SLOTS, 0) > 3:
objectives.append("Proactively offer waitlist when no immediate slots are available")
objectives.append("Suggest alternative dates before saying slots are unavailable")
if counts.get(FailureReason.CUSTOMER_DISENGAGED, 0) > 2:
objectives.append("Shorten the initial greeting to reach the customer's intent faster")
objectives.append("Use warmer, more engaging language to maintain customer attention")
if counts.get(FailureReason.AGENT_CONFIDENCE_LOW, 0) > 2:
objectives.append("Ask clarifying questions when the request is ambiguous")
objectives.append("Handle compound requests by addressing each part separately")
return objectives or ["Improve overall call resolution rate"]This function is where human expertise enters the system. Someone who understands voice agent failure modes wrote these mappings. NO_SLOTS failures suggest the prompt should handle slot unavailability more gracefully—that's domain knowledge, not something the system can infer automatically.
The objectives become part of the prompt sent to the LLM for generation. Instead of asking "make this prompt better," the system asks "make this prompt better by offering waitlists when slots are unavailable and asking clarifying questions when requests are ambiguous." Specific objectives produce specific improvements.
The Optimization Workflow
When Keep detects a metric anomaly—say, conversion rate dropping below 80%—it triggers a workflow that calls the optimizer. The workflow is defined in YAML and executed by Keep's workflow engine:
workflow:
id: voice-ai-remediation
triggers:
- type: alert
filters:
- key: name
value: conversion_rate_drop
steps:
- name: fetch-failures
provider:
type: postgres
with:
query: |
SELECT transcript, summary, failure_reason
FROM call_outcomes
WHERE success = false AND created_at > NOW() - INTERVAL '1 hour'
- name: trigger-optimization
provider:
type: http
with:
url: "{{ env.GEPA_OPTIMIZER_URL }}/optimize"
method: POST
body:
alert_id: "{{ alert.id }}"
failed_calls: "{{ steps.fetch-failures.results }}"The optimizer receives this payload and executes the generation workflow. It loads the current active prompt, derives objectives from the failure patterns, builds a generation prompt that includes the current prompt text plus failed call summaries plus objectives, calls the LLM to generate an improved version, and stores the result as a new prompt version.
The generation prompt looks like this:
Current prompt (version v3):
[current prompt text]
Recent failures (30 calls in the last hour):
- Customer asked about tire rotation, agent failed to classify intent
- Customer requested Tuesday instead of Monday, agent treated as new booking
- Customer mentioned husband's car, agent didn't handle vehicle mismatch
...
Objectives:
- Ask clarifying questions when the request is ambiguous
- Handle rescheduling requests distinctly from new bookings
- Confirm vehicle ownership before proceeding
Generate an updated prompt that addresses these failures while preserving what works.The LLM generates improved prompt text based on this context. The optimizer parses the response, stores it as a new version (v4), and makes it active. The next call to the agent will use the new prompt.
Prompt Versioning
Prompts aren't mutable text files. They're versioned artifacts with full history.
Every optimization creates a new version. Version identifiers increase monotonically: v1, v2, v3, and so on. The database stores the complete history: what each version's text contained, when it was created, why it was created (the optimization notes), and whether it's currently active.
@dataclass
class PromptVersion:
id: int
version: str # "v3"
content: str # Full prompt text
notes: str # "Addressed NO_SLOTS failures, added waitlist offer"
is_active: bool # Only one version is active at a time
created_at: datetimeVersioning enables several capabilities:
Rollback: If v4 performs worse than v3, you can revert by changing which version is active. The old text still exists; you're just pointing at it again.
Audit trail: Every prompt change has a recorded reason. "Why does the prompt mention waitlists?" Look at v4's notes: it was generated to address NO_SLOTS failures. This is essential for debugging and for explaining decisions to stakeholders.
Comparison: You can diff v3 against v4 to see exactly what changed. Did the optimizer add a new section? Modify existing wording? Remove something that was working? The version history makes this visible.
The versioning overhead is minimal—a few database rows per optimization run—but the debugging and rollback capabilities are invaluable. You never want to be in a situation where you can't recover a previous prompt because someone overwrote it.
Scoring Before Deployment
Not every generated prompt is an improvement. LLMs hallucinate. They misunderstand objectives. They sometimes make prompts worse while trying to make them better.
The system scores new prompts before deploying them. The score evaluates multiple dimensions:
Coverage: Does the prompt address the failure types that triggered optimization? If we optimized because of AGENT_CONFIDENCE_LOW failures, the new prompt should include language about handling ambiguous requests.
Specificity: Are the instructions concrete or vague? "Be helpful" is vague. "Ask clarifying questions when the customer's request mentions multiple services" is specific.
Brevity: Longer isn't always better. Prompts that balloon in size may confuse the model or hit token limits. The score penalizes excessive length.
@dataclass
class ScoreBreakdown:
coverage: float
specificity: float
brevity: float
def total(self) -> float:
return self.coverage + self.specificity + self.brevityThe score determines whether the prompt gets deployed. If score.total() exceeds a minimum threshold, the new version goes active. If not, the optimization run is logged as attempted but the previous prompt remains active.
This prevents regressions. A bad LLM generation won't tank your production system because it won't pass the scoring threshold. The safeguard isn't perfect—scoring is based on the prompt text, not actual call outcomes—but it catches obvious failures like prompts that ignore the optimization objectives entirely.
Why Keep?
You could build the detection piece yourself. Poll a metrics endpoint, fire webhooks when thresholds are breached, done. So why use Keep?
Keep provides value beyond simple threshold alerting:
Deduplication: If conversion drops across 15 dealerships simultaneously, you don't want 15 separate alerts. Keep correlates related events into a single incident. The optimizer gets called once, not 15 times.
Enrichment: Keep workflows can query multiple data sources, call external APIs, and enrich alerts with context before calling the optimizer. "Conversion dropped" becomes "Conversion dropped, 80% of failures are AGENT_CONFIDENCE_LOW, mostly happening in Texas region."
Lifecycle management: When metrics recover after a prompt update, Keep can auto-resolve the incident. The full remediation cycle—detection, diagnosis, fix, recovery—is tracked as one incident with clear timestamps for each phase.
Workflow orchestration: The workflow definition is declarative YAML, not scattered code. Changing the remediation logic means editing a config file, not modifying application code.
Using established AIOps tooling for ML systems is an underappreciated pattern. Prompt degradation is as much an operations problem as a model problem. The same techniques that monitor database latency and API error rates apply to voice agent conversion rates.
Limitations and Future Work
The system has clear limitations that I'd address with more time:
No shadow testing: New prompts deploy immediately to all traffic. A safer approach would deploy to 10% of calls first, compare conversion rates against the old prompt, and only fully roll out if the new version performs better in production.
Single-shot optimization: The optimizer generates one prompt per run. Iterative refinement—generate a candidate, test it, refine based on results, repeat—would likely produce better outcomes than single-shot generation.
Heuristic scoring: The scoring evaluates prompt text characteristics, not actual call outcomes. True validation requires A/B testing against live traffic, which the current system doesn't do.
Manual rollback: If a new prompt performs worse, reverting requires manual intervention. Automated rollback triggered by metric degradation would complete the self-healing loop.
These are engineering improvements, not fundamental architectural changes. The closed-loop structure is sound; the implementation could be more sophisticated.
Results
The complete remediation cycle works like this:
- Conversion rate drops below 80% → Keep fires alert
- Keep workflow fetches recent failed calls from database
- Workflow calls optimizer with failure data
- Optimizer derives objectives from failure distribution
- LLM generates updated prompt
- Scoring passes threshold; new version deployed
- Conversion recovers; Keep auto-resolves incident
Total cycle time: minutes, not hours.
Technology Stack
- Python 3.11, Flask for the optimizer API
- PostgreSQL for prompt version storage and call outcome logging
- Keep (open-source) for alerting and workflow orchestration
- Qwen via Together API for prompt generation
- Docker Compose for local development and testing