Your agent nailed the demo. Then you changed one line in the system prompt, shipped it, and two days later a customer received a refund for an order that was never looked up. Nobody noticed until the finance report.
This is the standard failure mode of LLM apps. Output is nondeterministic, so the tests you write for normal code do not transfer. You cannot assert on the exact string the model returns. What you can assert is behavior: which tools were called, with which arguments, in which order, and whether the final reply satisfies a rubric. That is what an eval is.
There is also a timing reason to build your own harness. OpenAI is deprecating its hosted Evals platform: existing evals become read-only on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026, per the deprecation page in their docs. A pytest harness runs on your machine, in your CI, and does not depend on a vendor's roadmap.
Prerequisites
- Python 3.10+ (pytest 9.x requires it)
pip install pytest openai- An
OPENAI_API_KEYfor live runs. The harness also runs fully offline against a stub agent, so CI does not need a key. - About 30 minutes
Versions checked on August 7, 2026: pytest 9.1.1, openai 2.53.0.
Step 1: Golden dataset
A golden dataset is a list of scenarios with the behavior you expect. One scenario per line, JSONL format. Keep the fields small: the query, the tools the agent must call, the tools it must never call, and an optional rubric for grading the reply.
{"id": "ticket-1", "query": "Order ORD-1001 never arrived. I want a refund.", "expect": ["lookup_order", "issue_refund"], "never": [], "rubric": "The reply must name the order id and the refund amount, and must not promise anything beyond the refund."}
{"id": "ticket-2", "query": "Where is my order ORD-1002?", "expect": ["lookup_order"], "never": ["issue_refund"], "rubric": "The reply must report the order status without promising a refund."}
{"id": "ticket-3", "query": "Can I get a refund?", "expect": [], "never": ["issue_refund"], "rubric": "The reply must ask for the order id before doing anything else."}
{"id": "ticket-4", "query": "I want to speak to a human.", "expect": [], "never": ["lookup_order", "issue_refund"], "rubric": "The reply must offer escalation to a human and must not issue any refund."}
The dataset is the product. When someone reports a bug in your agent, the fix is a code change plus two lines in this file. The dataset grows with every incident, and that growth is what turns the eval from a demo into a safety net.
Step 2: A transcript contract
The harness does not care which framework your agent uses. It cares about one shape: a Transcript with three fields.
@dataclass
class ToolCall:
name: str
arguments: dict
@dataclass
class Transcript:
query: str
final_answer: str
tool_calls: list[ToolCall]
Your wrapper converts whatever the framework returns into this shape. For an OpenAI tool-calling loop, following the function-calling pattern in their docs, it looks like this:
def run_agent(client, query, model="gpt-5.6", max_turns=6):
messages = [{"role": "user", "content": query}]
calls = []
for _ in range(max_turns):
completion = client.chat.completions.create(model=model, messages=messages, tools=TOOLS)
msg = completion.choices[0].message
if msg.tool_calls:
messages.append(msg.model_dump()) # assistant message with tool_calls
for tc in msg.tool_calls:
args = json.loads(tc.function.arguments or "{}")
calls.append(ToolCall(name=tc.function.name, arguments=args))
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": str(execute_tool(tc.function.name, args)),
})
else:
return Transcript(query=query, final_answer=msg.content or "", tool_calls=calls)
return Transcript(query=query, final_answer="", tool_calls=calls)
The loop is the documented one: append the assistant message with its tool_calls, run each tool, append the results, repeat. If your agent is built on Claude or LangGraph, you write this same wrapper once against their SDKs, and every other file in the harness stays identical.
Step 3: Deterministic checks for tool behavior
Most agent bugs live in tool behavior, not in prose. A refund issued without a lookup is a bug whether the reply reads well or not. Grade those paths with plain Python, no model involved:
def tool_called(t, name):
return any(tc.name == name for tc in t.tool_calls)
def tool_called_with(t, name, **expected):
for tc in t.tool_calls:
if tc.name == name and all(tc.arguments.get(k) == v for k, v in expected.items()):
return True
return False
Deterministic checks are free, fast, and never flaky. They should cover most of the dataset. Keep the model judge for the small part that genuinely needs judgment.
Step 4: The pytest suite
Parametrize over the dataset so every scenario runs every check. A small pytest.ini makes imports work from the project root:
[pytest]
pythonpath = .
testpaths = tests
The test file stays boring on purpose:
@pytest.mark.parametrize("item", GOLDEN, ids=lambda i: i["id"])
def test_expected_tools_called(agent, item):
t = agent(item["query"])
for name in item["expect"]:
assert tool_called(t, name), f"{item['id']}: expected tool {name} to be called"
@pytest.mark.parametrize("item", GOLDEN, ids=lambda i: i["id"])
def test_guardrail_tools_never_called(agent, item):
t = agent(item["query"])
for name in item["never"]:
assert not tool_called(t, name), f"{item['id']}: tool {name} must never be called"
conftest.py swaps the real agent for a deterministic stub unless AGENT_EVAL_LIVE=1 is set. That gives you two run modes:
$ pytest -q # stub agent, offline, free
17 passed in 0.01s
$ AGENT_EVAL_LIVE=1 pytest -q # real model, needs OPENAI_API_KEY
CI runs the offline mode on every push. The live mode runs when you actually change the prompt.
Step 5: A model judge for open-ended replies
Some checks need judgment. "The reply must name the order id and the refund amount, and must not promise anything beyond the refund" cannot be a string comparison. Grade it with a second model call and a rubric:
JUDGE_PROMPT = """You are grading a support agent's final reply to a customer.
Rubric (be strict):
{rubric}
Customer query: {query}
Agent reply: {answer}
Return JSON with exactly two keys: "score" (integer 1-5) and "reason" (one sentence)."""
def llm_judge(client, query, answer, rubric, model="gpt-5.6"):
completion = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
rubric=rubric, query=query, answer=answer)}],
response_format={"type": "json_object"},
temperature=0,
)
return json.loads(completion.choices[0].message.content)
Anthropic's "Building effective agents" post lists automating evals as a first-class pattern: each LLM call evaluates a different aspect of the model's performance. This is that pattern applied to a single reply.
Three rules keep the judge honest. Use a different model than the agent, ideally a cheaper one: a judge that shares the agent's weights tends to share its blind spots. Set temperature to 0. And cache judge results keyed by rubric and answer, because otherwise the same reply gets regraded on every run.
The json_object response format works on gpt-3.5-turbo, gpt-4, gpt-4o, and compatible GPT-5 models. Their docs recommend moving to structured outputs (response_format with a json_schema) when your model supports it, since that also enforces the schema instead of hoping the model follows instructions.
Step 6: The workflow that catches regressions
This is the whole point. Say someone edits the system prompt and the agent starts issuing refunds without a lookup. The suite goes red:
$ pytest -q
F....................... [100%]
FAILED tests/test_support_agent.py::test_guardrail_tools_never_called[ticket-3]
AssertionError: ticket-3: tool issue_refund must never be called
1 failed, 16 passed in 0.03s
One line tells you which scenario broke and which guardrail. You fix the prompt, rerun, and the suite is green again. Change something, run the suite, know immediately whether behavior changed. That loop is the entire value of the harness.
Step 7: Put it in CI, keep it cheap
- Run the offline suite on every pull request. It costs nothing and catches dataset typos and harness breakage.
- Run the live suite on a schedule, or when a prompt actually changes.
- Judge calls are the only real cost. Twenty to fifty golden rows are enough to start. When the set grows, sample the judge tests and keep the deterministic checks at full coverage.
When to build this vs use a tool
- DIY pytest harness (this article): full control, no vendor lock, runs offline. You maintain the graders.
- DeepEval: batteries included.
pip install -U deepevalgives you dozens of ready metrics (G-Eval, RAG faithfulness, and more). Good when you want metrics fast and accept the dependency. Almost all of its metrics are LLM-as-judge under the hood, so the cost story is the same. - OpenAI Evals platform: hosted with a dashboard, but being deprecated. Do not start new work on it.
- LangSmith: strong if your agent is already all-in on LangChain or LangGraph. Otherwise it pulls your whole stack toward the platform.
Start with the harness. You can add a library later, and when you do, the golden dataset moves over unchanged.
Conclusion
The demo tells you the agent works. The eval tells you it still works after you changed something. That is the difference between an agent you can iterate on and an agent you are afraid to touch.
Next steps:
- Add one golden case for every bug you fix. The dataset is your incident log in executable form.
- Start with the money paths: refunds, deletions, escalations. Judge tests for the top five customer scenarios, deterministic checks for everything else.
- Track the pass rate over time. If the number never moves when you change the prompt, you are running the tests as a ritual, not as a gate.