A prompt that returns clean JSON from one model can fall apart on the next one. You add three lines of "IMPORTANT: return valid JSON", ship it, and the failures move to a different batch of tickets. Two rounds of that, and the prompt is a stack of special cases nobody wants to touch.
There is a duller way to fix it. Define a metric over your own labeled examples, then let an optimizer rewrite the instruction text until the score stops improving. That is what GEPA does inside DSPy, and this guide covers the full loop on a small ticket-triage program: baseline, metric with feedback, optimization, save, then serving the optimized program on a local model through Ollama.
Prerequisites
- Python 3.10 or newer
- dspy 3.3.1 (
pip install dspy, oruv add dspy) - 40 to 100 labeled examples for your task. Small is fine; the budget math further down shows the cost
- An API key for a strong model to use as the reflection model
- Optional: Ollama running on localhost:11434, if the model you deploy runs locally
Every API call in the code below was run against dspy 3.3.1 to confirm the signatures.
Step 1: Write the program, not the prompt
A DSPy program is a class with declared inputs and outputs. The docstring on the signature becomes the instruction that GEPA rewrites.
# triage.py
import dspy
class Triage(dspy.Signature):
"""Route a support ticket to the right team and set its priority."""
ticket: str = dspy.InputField()
category: str = dspy.OutputField(desc="billing, bug, or howto")
priority: str = dspy.OutputField(desc="low, normal, or urgent")
class TriageProgram(dspy.Module):
def __init__(self):
super().__init__()
self.classify = dspy.Predict(Triage)
def forward(self, ticket):
return self.classify(ticket=ticket)
Two output fields, both short strings. Keep that output schema frozen for the whole optimization. If the optimizer is allowed to change what the task is, it will.
Step 2: Label a small dataset
train = [
dspy.Example(ticket="I was charged twice this month.", category="billing", priority="urgent"),
dspy.Example(ticket="How do I export my notes?", category="howto", priority="low"),
# ... 30 to 60 more
]
val = [
dspy.Example(ticket="App crashes when I hit export.", category="bug", priority="normal"),
# ... a separate set, held out from training
]
trainset = [ex.with_inputs("ticket") for ex in train]
valset = [ex.with_inputs("ticket") for ex in val]
with_inputs("ticket") marks the field the model sees. The labels stay behind as ground truth. Keep the two sets separate. Pass the same rows as both trainset and valset and DSPy logs a warning for a reason: the optimizer fits instructions to those exact rows, and you ship a prompt that only works on your own examples.
Step 3: Write a metric that explains failures
This is what separates GEPA from an optimizer that only sees scores. The metric can return text, and GEPA feeds that text into the reflection prompt verbatim.
def triage_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
got_cat = (pred.category or "").strip().lower()
got_pri = (pred.priority or "").strip().lower()
score = (int(got_cat == gold.category) + int(got_pri == gold.priority)) / 2
feedback = (
f"Predicted category={got_cat}, expected {gold.category}. "
f"Predicted priority={got_pri}, expected {gold.priority}. "
"Write a general rule that covers tickets like this one. Do not copy "
"this example's words into the instructions."
)
return dspy.Prediction(score=score, feedback=feedback)
DSPy calls this twice per evaluated example: once for the program as a whole, and once per predictor while that predictor is being rewritten. One function handles both calls.
Step 4: Get the baseline number
import os
import dspy
from triage import TriageProgram
from triage_metric import triage_metric, trainset, valset
student_lm = dspy.LM("openai/gpt-5-nano", api_key=os.environ["OPENAI_API_KEY"])
dspy.configure(lm=student_lm)
program = TriageProgram()
evaluate = dspy.Evaluate(
devset=valset,
metric=triage_metric,
num_threads=8,
display_progress=True,
display_table=False,
)
baseline = evaluate(program)
print(f"baseline score: {baseline.score:.1f}%")
dspy.Evaluate returns an EvaluationResult and .score is already a percentage. Note the student model here is the cheap one you plan to ship, not the best model you have access to. Optimizing for a model you will not deploy is a waste of a budget.
Step 5: Run GEPA
reflection_lm = dspy.LM("openai/gpt-5.4", temperature=1.0, max_tokens=32000)
optimizer = dspy.GEPA(
metric=triage_metric,
reflection_lm=reflection_lm,
auto="light",
num_threads=2,
track_stats=True,
)
optimized = optimizer.compile(program, trainset=trainset, valset=valset)
optimized_eval = evaluate(optimized)
print(f"optimized score: {optimized_eval.score:.1f}%")
Three things worth knowing before you press enter.
- A reflection model is mandatory. Skip
reflection_lmand GEPA raises an assertion immediately rather than halfway through a paid run. The reflection model reads failing traces and proposes new instructions, so give it something strong. It gets called a handful of times, so the cost stays small. autois the budget. The settings map to 6, 12, and 18 candidate prompts for light, medium, and heavy. On a one-predictor program with a 40-example valset,auto="light"works out to 540 metric calls, roughly 13.5 full evaluations of the program. Medium is 890, heavy is 1315. A 100-example valset scales those numbers up in proportion.num_threadsspeeds up the student, not the reflection step. Reflective proposals run serially by design, so two threads is a reasonable default while you watch your rate limits.
Step 6: Save the result and read the diff
optimized.save("triage_optimized.json")
fresh = TriageProgram()
fresh.load("triage_optimized.json")
print(fresh.classify.signature.instructions)
The optimizer's output is instruction text living in each predictor's signature, so save() writes a JSON file you can diff in a pull request. A few documented behaviors worth remembering:
api_keyis never serialized. The loading side configures credentials on its own.api_base,base_url, andmodel_listget stripped on load unless you passallow_unsafe_lm_state=True.- A
.pklsave or a full-program save requiresallow_pickle=Trueto load, because deserializing a pickle can execute code. State-only JSON carries no such risk, which is why it is the default recommendation.
Step 7: Serve the optimized program on a local model
The workflow that actually saves money is asymmetric. Optimize with a frontier model as the reflection model, then run the optimized program on a small model you host yourself.
dspy.configure(
lm=dspy.LM("ollama_chat/qwen3:8b", api_base="http://localhost:11434", api_key="")
)
Prefer the ollama_chat/ prefix over ollama/. LiteLLM's docs recommend it for better responses, since it routes through the chat completions path instead of raw completion. DSPy normalizes model strings through LiteLLM, so every provider LiteLLM supports works the same way here.
Where this pays off, and where it bites
Use published results as sanity checks, not as promises. Dropbox ran GEPA on their relevance judge while moving from o3 to an open-weight model: NMSE against human ratings dropped 45 percent (8.83 to 4.86), model adaptation went from one to two weeks of manual iteration down to one to two days, and the cheaper judge let them label 10 to 100 times more data at the same cost. The DSPy walkthrough reports a student model climbing from 78.1 percent to 90.1 percent on its haiku task after GEPA, past the 82.4 percent baseline of the unoptimized frontier model. Both numbers belong to someone else's task. Your own valset is the only number that counts.
Two failure modes show up in practice.
The first is feedback that invites memorization. Dropbox watched candidate prompts absorb specific usernames and verbatim document phrases. Those candidates scored well on training rows and generalized badly. The fix lives in the feedback string: instruct the reflection model to write rules, and to keep example-specific words out of the instructions. If your task has a fixed label set or a rating scale, say outright that those cannot change. Dropbox saw candidates quietly narrow a 1 to 5 scale down to 1 to 3, which breaks every comparison downstream.
The second is optimizing against the rows you will be judged on. Without a held-out valset you are measuring fit on the traffic you labeled, not generalization to new tickets. GEPA's Pareto sampling is more forgiving than greedy search, since it keeps candidates that win on at least one validation example, but no optimizer can invent signal your data does not contain.
When to use GEPA versus the alternatives
| Situation | Better fit |
|---|---|
| Under 20 labeled examples, no metric defined yet | A hand-written prompt. Label data first |
| Plenty of examples, failures that are hard to put into words | BootstrapFewShot or MIPROv2 |
| Failures you can explain in a sentence, and rollouts that cost money | GEPA |
| Millions of cheap rollouts, and the model weights should change too | Fine-tuning or RL, optionally after prompt optimization |
GEPA is built for the case where each rollout costs something and textual feedback is available, which describes most production LLM tasks. Keep BootstrapFewShot in mind when all you actually need is better few-shot demos.
Next steps
Start with 40 labeled examples, a scoring metric, and auto="light". Record the baseline before you optimize, or you will never know whether the optimizer earned its budget. Commit the saved JSON so prompt history lives in git alongside the code. Then run the same loop the next time you want to swap in a cheaper model, and let your valset decide whether the swap is safe.