Mini-Town was an ambitious experiment that I eventually abandoned. The goal was to build an agent-based simulation where autonomous characters—powered by LLMs—would perceive their environment, form memories, reflect on experiences, and generate plans. Instead of hand-tuning prompts, I would use DSPy's GEPA optimizer to automatically improve the agents' "brains" based on collected data.
The project reached a functional state. Agents moved around a 2D world, scored observations for importance, stored memories with vector embeddings, and generated daily plans. I got a 17% improvement in observation quality using compiled prompts versus hand-written ones. But the complexity grew faster than the value, and I ultimately set it aside.
This post isn't a tutorial on how to build what I built. It's a collection of the things I learned along the way—lessons that transferred to other projects and shaped how I think about LLM-based systems.
The Original Idea
The project emerged from a simple question: can you treat LLM prompts as programs that can be automatically optimized?
When you hand-write a prompt, you're making guesses about what wording will produce the behavior you want. "Rate this observation from 1-10 based on importance" might work, but "Evaluate the significance of this event on a scale of 1-10, where 10 means life-changing and 1 means completely mundane" might work better. Or worse. You don't know until you try, and trying is expensive and slow.
DSPy offers an alternative. Instead of writing prose instructions and hoping for the best, you define a typed signature—a formal specification of what inputs the LLM receives and what outputs it should produce. The framework then uses training examples to search for prompts that produce correct outputs. It's treating prompt engineering as an optimization problem rather than an art form.
The approach is compelling because it scales differently than manual iteration. If you have 50 training examples and can afford 1000 LLM calls during optimization, the optimizer can explore far more prompt variations than a human could in a week. And because the optimization is metric-driven, it's less susceptible to the subtle biases humans bring to prompt writing.
Mini-Town was the testbed for this idea. Build a simulation with multiple agents, log their cognitive outputs, use those logs as training data, and compile improved prompts. The hypothesis: compiled agents would behave more consistently and intelligently than uncompiled ones, and we could measure the difference.
The simulation itself was inspired by Stanford's "Generative Agents" paper—the one with the AI town where characters formed relationships, remembered conversations, and threw parties. I wasn't trying to replicate that research; I was using the simulation as a vehicle to test prompt optimization. The town was scaffolding. The real experiment was the compilation.
What Got Built
The architecture had three main components, each representing a different layer of the system.
┌─────────────────────────────────────────────────────────────────────────────┐
│ SIMULATION ENVIRONMENT │
│ │
│ ┌────────────────────┐ ┌───────────────────────────────────────────┐ │
│ │ 2D World (Grid) │ │ Agent Population │ │
│ │ ───────────── │ │ ───────────────── │ │
│ │ • Locations │◀────▶│ Alice, Bob, Maria, ... │ │
│ │ • Entities │ │ Each running cognitive loop │ │
│ │ • Time (ticks) │ │ │ │
│ └────────────────────┘ └──────────────────┬────────────────────────┘ │
└─────────────────────────────────────────────────┼────────────────────────────┘
│ perceive environment
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ COGNITIVE LOOP (per agent) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │
│ │ PERCEIVE │───▶│ SCORE │───▶│ STORE │───▶│ REFLECT │───▶│ PLAN │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ Detect │ │ Rate 1-10│ │ Embed + │ │ Synthesize│ │ Daily │ │
│ │ nearby │ │ importance│ │ save to │ │ insights │ │ schedule│ │
│ │ entities │ │ (DSPy) │ │ memory │ │ (DSPy) │ │ (DSPy) │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └────────┘ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ ▼ │
│ │ ┌─────────────────────────────────────────────────────┐ │
│ │ │ DSPy Modules │ │
│ │ │ ┌────────────────┐ ┌───────────────┐ ┌─────────┐ │ │
│ │ │ │ ScoreImportance│ │ ReflectOnDay │ │ MakePlan│ │ │
│ │ │ │ (typed sig) │ │ (typed sig) │ │ (typed) │ │ │
│ │ │ └────────────────┘ └───────────────┘ └─────────┘ │ │
│ │ │ │ │
│ │ │ GEPA Optimizer: Compiles signatures → better prompts│ │
│ │ └─────────────────────────────────────────────────────┘ │
└───────┼─────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ MEMORY SYSTEM (DuckDB + VSS) │
│ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Weighted Retrieval: score = α×relevance + β×recency + γ×importance │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Relevance │ │ Recency │ │ Importance │ │ │
│ │ │ (cosine │ │ (time decay │ │ (score at │ │ │
│ │ │ similarity)│ │ function) │ │ storage) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ │
│ Embeddings: all-MiniLM-L6-v2 (384-dim, local) │
└─────────────────────────────────────────────────────────────────────────────┘The Cognitive Loop
Each agent ran a perceive-score-store-reflect-plan cycle that mimicked the approach from the Generative Agents paper, adapted for DSPy's typed program model.
Perception: On every simulation tick (every 2 seconds), agents would detect nearby entities. If Alice walked past Bob, Bob's perception system would register "Alice is nearby" as a raw observation. The perception range was configurable—agents could "see" anything within a certain radius.
Scoring: Raw observations aren't all equally important. Seeing a tree is mundane. Seeing someone collapse is urgent. The scorer was a DSPy module that took an observation and the agent's current context (their personality, goals, recent thoughts) and produced an importance score from 1-10. High scores triggered downstream processing; low scores were stored but didn't prompt reflection.
Storage: Every scored observation went into the memory database. The storage included not just the text and score, but also a vector embedding and a timestamp. This enabled the retrieval system to find memories by meaning, not just by recency.
Reflection: When accumulated importance exceeded a threshold, the agent would pause to reflect. The reflector was another DSPy module that took recent memories, retrieved semantically similar older memories, and synthesized a high-level insight. "I've noticed Maria keeps mentioning her garden" might become "Maria is passionate about gardening—I should ask her about it." Reflections themselves were stored as high-importance memories.
Planning: Periodically (configurable, but typically once per simulated "day"), agents would generate a schedule. The planner received the agent's goal, relevant memories, and any pending invitations, and produced a time-blocked plan: "7am: wake up, 8am: go to cafe, 10am: visit Maria's garden, 12pm: lunch at diner."
The loop ran continuously. Every tick, agents perceived. Interesting perceptions got scored. Accumulating importance triggered reflection. Time passing triggered planning. Actions came from plans. The result was agents that appeared to make decisions, though everything was actually just LLM calls wrapped in state management.
The Memory System
Memory in agent simulations is deceptively complex. Humans don't just remember things; they remember things that are relevant to their current situation, weighted by how recent they are and how important they seemed when they happened. Implementing that requires more than a list sorted by timestamp.
I used DuckDB with the Vector Similarity Search (VSS) extension. DuckDB gave me SQL for structured queries, and VSS gave me approximate nearest-neighbor search over embedding vectors. The combination let me implement weighted retrieval:
score = α × relevance + β × recency + γ × importanceRelevance was cosine similarity between the query embedding (what the agent was currently thinking about) and the memory embedding. This made semantic recall possible. An agent thinking about "parties" could retrieve a memory containing "Maria's get-together" even though the word "party" never appeared.
Recency was an exponential decay based on how long ago the memory was formed. Recent memories scored higher. The decay rate was tunable; faster decay made agents forgetful, slower decay gave them longer attention spans.
Importance was the score assigned at storage time. Salient memories—the ones that felt significant when they happened—remained accessible longer than mundane observations.
The weights (α, β, γ) controlled the behavior. High α meant agents prioritized semantic relevance (useful for focused tasks). High β meant agents focused on recent events (useful for reactive behavior). High γ meant agents surfaced important moments regardless of recency (useful for long-term relationship tracking).
Getting these weights right was one of the harder engineering problems in the project. I'll return to this later.
The DSPy Modules
Each cognitive function was implemented as a DSPy program with a typed signature. Signatures define the inputs and outputs formally, which lets the optimizer understand what it's trying to improve.
The scorer signature looked like this:
class ScoreImportance(dspy.Signature):
"""Score how important an observation is to this agent."""
observation: str = dspy.InputField(desc="What the agent just observed")
agent_personality: str = dspy.InputField(desc="The agent's personality traits")
agent_goal: str = dspy.InputField(desc="The agent's current goal")
recent_context: str = dspy.InputField(desc="What the agent was just thinking about")
score: int = dspy.OutputField(desc="Importance score from 1-10")
reasoning: str = dspy.OutputField(desc="Why this observation matters or doesn't")The reflector and planner had similar structures. Each defined what went in and what came out, with human-readable descriptions that helped both the optimizer and the developer understand the expected behavior.
The baseline implementation wrapped these signatures in dspy.ChainOfThought, which prompted the LLM to reason step-by-step before producing outputs. Chain of thought improved accuracy on the importance scoring task, at the cost of additional latency and tokens.
The compiled implementation used GEPA-optimized prompts that achieved similar or better accuracy with fewer tokens. More on compilation in the next section.
The Frontend
The frontend was a Next.js application that connected to the backend via WebSocket. It rendered a 2D map showing agent positions, displayed speech bubbles when agents "spoke" their plans or observations, and provided a diagnostic panel for inspecting agent state.
The visual representation was mostly for debugging. Watching an agent walk in circles told me its pathfinding was broken. Watching an agent stand still told me its LLM calls were timing out. Watching two agents never interact told me the perception radius was too small.
Most development happened headless (running the backend with logging and no frontend), but the visual mode was invaluable for the moments when logs didn't explain what was happening.
The Compilation Experiment
The core experiment was simple: could GEPA find better prompts than I could write by hand?
GEPA (Generalized Evolutionary Prompt Acceleration) is a DSPy optimizer that uses evolutionary strategies to search for improved prompts. You provide a training set of input-output examples (called "seeds"), a metric function that scores outputs, and a search budget. GEPA generates prompt variations, evaluates them against the training set, and iteratively improves toward the best-performing variants.
The process took about 15 minutes on a T4 GPU (Google Colab free tier). Running 80 optimization iterations over 40 seed examples, GEPA evaluated roughly 3000 prompt variants to find the best-performing one. The result was a compiled scorer that outperformed my hand-written baseline.
In a controlled 5-minute A/B test:
- Uncompiled agents generated 38 observations, avg quality score 0.67
- Compiled agents generated 53 observations, avg quality score 0.78
That's a 17% improvement in observation quality and 39% more observations in the same time. The compiled scorer was also 46% faster in isolation tests (0.46s vs 0.85s latency) because the optimized prompt was more concise.
The reflection and planning modules couldn't be compiled as easily because their outputs were harder to evaluate. What makes a "good" reflection? The question is inherently subjective. I could score importance reasonably consistently (a 7 is more significant than a 3), but I couldn't score reflections or plans reliably enough to train an optimizer.
This asymmetry—some cognitive functions are easy to evaluate, others are hard—limited how far compilation could take the project.
Why I Abandoned It
Several factors converged to make the project not worth continuing.
The Evaluation Problem
How do you measure whether an agent simulation is "good"?
I defined metrics. Observation quality was scored by an LLM judge. Plan coherence was measured by whether the plan's activities matched the agent's stated goals. Event attendance was measured by whether agents showed up to parties they'd been invited to.
But these were proxies for something I couldn't directly measure: whether the agents were interesting to watch. The 17% improvement in observation quality didn't translate into 17% more interesting behavior. It was an improvement on a metric that didn't capture what mattered.
The fundamental problem with open-ended simulations is that "good behavior" is defined by human intuition, not by formal criteria. When Alice decides to visit the park, is that good or bad? It depends on what Alice has been doing, what her goals are, who else is at the park, what time of day it is, and—critically—what the observer finds interesting. I couldn't formalize any of that.
I tried using "town score," a weighted combination of observation quality, reflection depth, plan coherence, and event attendance. The town score improved with compilation. But watching an agent with a high town score felt the same as watching an agent with a medium town score. The metric didn't correlate with the experience.
The Cost Problem
Even using cheap models (Llama 3.2 on Together.ai at $0.20 per million tokens), the API costs added up quickly when running multiple agents continuously.
Each tick involved multiple LLM calls: scoring observations, occasionally reflecting, checking if plans needed updating. With 5 agents and a 2-second tick, the simulation made roughly 250 LLM calls per minute. At ~500 tokens per call (including prompt and response), that was 125,000 tokens per minute, or about $1.50 per hour.
That sounds cheap, but I needed sustained simulation to generate training data. GEPA optimization required a corpus of logged examples, which meant running the simulation for hours to collect data before I could even start the optimization process. Then I needed to run the simulation again to A/B test the results. Development required many such cycles.
The Complexity Problem
The cognitive loop introduced cascade failures that were difficult to debug and impossible to prevent.
If the scorer ran slowly (maybe the API was throttled, maybe the model was overloaded), observations accumulated faster than they could be processed. The observation queue grew. Old observations became stale. Reflections based on stale observations were nonsensical.
If reflection failed (maybe the LLM produced an unparseable response), the agent's accumulated importance wasn't reset. It would try to reflect on the same observations again next tick, fail again, and enter an infinite loop of failed reflections.
If planning failed, the agent defaulted to random walking. But random walking generated more observations, which triggered more scoring and more reflection, which increased load on the LLM, which made planning more likely to fail due to rate limits.
Each component's failure propagated to everything downstream. Debugging required understanding the interaction of five different subsystems, each with its own state and failure modes. I spent more time building monitoring and debugging tools than I spent building the actual simulation.
The Ambiguity Problem
Agent behavior in open-ended simulations is inherently ambiguous. This made creating training data painful.
GEPA requires seed examples: input-output pairs where the output is the "correct" answer. For the scorer, I could produce these. "Observation: Maria invited me to her party. Score: 8. Reason: Social invitation with future commitment." That's defensible.
But what about edge cases? "Observation: I see a tree." Is that a 1 or a 2? Depends on whether the agent cares about nature. "Observation: Bob walked past without saying hello." Is that significant? Depends on whether Bob usually says hello. "Observation: The sun is setting." Is that important? Depends on whether the agent needs to get home before dark.
I created an annotation guide to standardize scoring, then had a friend score some examples independently to check agreement. Cohen's kappa was 0.65—substantial agreement, but not perfect. The residual disagreement meant the training data had noise, which limited how much GEPA could improve.
Better annotation would have required more annotators, more examples, and a more rigorous process. At that point, I was building a labeling pipeline, not an agent simulation.
What I Learned
Despite abandoning the project, it shaped how I approach LLM systems now.
Structure Your Outputs from the Start
Early versions of the cognitive loop logged free-form text. "Agent observed something interesting" tells you nothing when you're trying to debug why an agent walked into a wall.
Later versions logged structured data:
- Observation content (what the agent saw)
- Timestamp (when it happened)
- Importance score (how significant it was)
- Scoring latency (how long the LLM took)
- Failure reason (if scoring failed, why)
Structured outputs made debugging possible. When an agent started behaving erratically, I could query the logs: "Show me all observations with score > 7 from the last 5 minutes." I could trace causality: "This high-importance observation triggered reflection at 10:42, which produced this insight, which affected the plan generated at 10:45."
Structured outputs also made training data collection tractable. Each logged item had the fields GEPA expected: input (observation + context) and output (score + reasoning). I didn't have to post-process logs into training examples; they were already in the right format.
If I'd started structured from day one, I would have saved days of refactoring. Every project I've started since then begins with a schema for the data it will log.
Measure Latency Before Complexity
I measured latency on Day 2 specifically to avoid a trap: building a complex system that couldn't run in real time.
The process was simple. I wrapped every LLM call in a timing function:
async def timed_llm_call(func, *args, **kwargs):
start = time.perf_counter()
result = await func(*args, **kwargs)
elapsed = time.perf_counter() - start
latency_tracker.record(func.__name__, elapsed)
return resultAfter a few minutes of simulation, I had a latency distribution. The scorer's p50 was 0.8 seconds; p95 was 1.4 seconds. The reflector's p50 was 1.2 seconds; p95 was 2.1 seconds.
The simulation tick was 2 seconds. That meant the cognitive loop had to complete its per-tick work (scoring + maybe reflecting) in under 2 seconds, or the simulation would lag. With scoring alone taking 1.4 seconds at p95, I had 0.6 seconds of headroom. Reflection was out of the question on every tick; it would have to happen on a slower schedule.
This constraint informed every design decision. I couldn't add a second LLM call per tick. I couldn't make prompts longer (more tokens = more latency). I couldn't use larger models. The latency ceiling was load-bearing.
In later projects, I measure latency first, before adding features. You can always add complexity; you can't always remove latency.
Compilation Works, but Training Data Is Hard
GEPA did what it promised: given training examples, it found better prompts. The 17% improvement was real. The optimized prompts were shorter and faster. The mechanism works.
But creating training examples required human judgment about what "correct" meant, and that judgment was expensive to exercise consistently. For the scorer, I spent three hours annotating 40 seed examples. Each example required reading an observation, imagining the agent's context, deciding on a score, and writing a rationale. Forty examples was enough to show improvement but not enough to reach the optimizer's full potential. GEPA's documentation suggests 200+ examples for best results.
Generating 200 examples at 4 minutes per example would take 13 hours. That's a full weekend of labeling. And if my scoring criteria evolved during that time (which they would, because annotating forces you to clarify your standards), earlier examples would be inconsistent with later ones.
For tasks with clear correctness criteria (classification, extraction, structured output), DSPy compilation is worth the effort. You can generate training data programmatically or collect it from production usage. But for tasks where correctness is subjective (open-ended generation, creative writing, behavioral simulation), the training data problem dominates everything else.
Memory Retrieval Is Surprisingly Finicky
The vector search used a weighted combination: relevance (semantic similarity), recency (time decay), and importance (score at storage time). Getting the weights right required extensive tuning.
In early tests, I set the weights to α=1.0, β=0.0, γ=0.0—pure relevance. The system seemed to work. Agents retrieved memories that were semantically related to their current situation. Great.
Then I tested a scenario where an agent needed to remember what happened earlier that day. Pure relevance failed. The agent retrieved a semantically similar but weeks-old memory instead of the relevant recent one. I needed recency.
Then I tested a scenario where an agent needed to remember an important conversation from last month. With high recency weight, that memory was suppressed by mundane recent observations. I needed importance.
Different scenarios needed different weights:
- Emergencies: high relevance (α=0.7), find anything about this topic
- Social planning: high recency (β=0.6), what happened today
- Long-term relationships: high importance (γ=0.7), filter out noise
I built a grid search to find optimal weights per scenario type, but the scenarios themselves were hard to define rigorously. The retrieval system worked, but tuning it was a research project in itself.
The lesson: test retrieval with adversarial examples, not just the happy path. If you only test with data where relevance and recency correlate (recent events that happen to be semantically related), you'll think your system works when it's actually ignoring half its inputs.
Provider Flexibility Saves Projects
The project nearly stalled on Day 5 when Groq's free tier hit rate limits mid-experiment.
I was running an A/B comparison: 20 minutes of uncompiled simulation, then 20 minutes of compiled simulation, then compare the logs. Halfway through the uncompiled run, Groq started returning 429 errors. Rate limited. The remaining observations went unscored, ruining the comparison.
The fix was switching to Together.ai, which had no rate limits on their serverless models. But I'd hardcoded Groq as the provider. DSPy was configured to use Groq. The API key was a Groq key. Changing providers required refactoring the configuration to support multiple backends.
I added a config file:
llm:
provider: together # or groq, or openai
model: meta-llama/Llama-3.2-3B-Instruct-Turbo
api_key: ${TOGETHER_API_KEY}And a dispatch function:
def configure_dspy():
config = load_config()
if config.llm.provider == "together":
lm = dspy.LM("together_ai/" + config.llm.model)
elif config.llm.provider == "groq":
lm = dspy.LM("groq/" + config.llm.model)
# ... etc
dspy.configure(lm=lm)After that, switching providers was a one-line config change. The experiment restarted in minutes.
Every project I build now has provider abstraction from day one. The free tier that works today will hit limits tomorrow. The model that's fastest now will be deprecated next month. The only constant is change, and the only defense is abstraction.
What Remains
The code still runs. The frontend renders agents moving through a 2D world. The cognitive loop still scores observations, stores memories, and generates plans. The compiled scorer still outperforms the baseline.
But I don't work on it anymore. The lessons transferred to other projects:
- The voice agent optimization system uses similar memory structures for storing failed calls
- The fraud detection system uses structured failure logging to enable downstream analysis
- The deal research agent uses provider abstraction for resilient LLM access
- Everything I build now measures latency before adding complexity
Mini-Town was the laboratory where I figured out how to build those things. The experiment failed to produce a compelling simulation, but it succeeded at teaching me how to build robust LLM systems.
Sometimes the value of a project isn't in what it produces. It's in what it teaches you.
Technical Details
For those interested in the specifics:
- Backend: FastAPI (Python 3.11), DuckDB with VSS extension for vector search
- Frontend: Next.js/React with TailwindCSS, WebSocket for real-time updates
- LLM: Together.ai's Llama 3.2-3B-Instruct-Turbo (serverless, ~$0.20/M tokens)
- DSPy: GEPA optimizer for prompt compilation, ChainOfThought for base modules
- Embeddings: all-MiniLM-L6-v2 (384-dimensional, runs locally)
- Compilation budget: 40 seed examples, 80 GEPA iterations, ~15 minutes on T4 GPU
The 17% improvement came from comparing observation quality scores (LLM-judged) between compiled and uncompiled runs over 5-minute simulations with 3 agents. The compiled scorer was tested in isolation showing 46% latency reduction (0.46s vs 0.85s).
Total development time was approximately 5 days of focused work, spread over several weeks. Total API cost was under $5, mostly consumed during the final A/B testing phase.
Mini-Town is archived on GitHub. The code works for what it was built to do—it's just not something I'm building further.