Turn a failed agent run into an eval that blocks the release
You fixed a bad agent run, but nothing stops the bug from coming back. By the end of this guide you can:
- Save one failed agent run as an eval case that you can replay.
- Grade what changed in the system, not what the agent said.
- Run the cases in pytest so a regression stops the release.
You need Python 3 and pytest. No API key and no network.
Before you start
This is episode 2. You should know Python functions, dicts and the json module, how a tool-calling loop works and how to write a pytest test. If not, open these first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.
- Debug an LLM agent: find the first bad step in its trace. Episode 1. We reuse its refund run, and the recap below is short.
- How an LLM tool-calling loop works. The model asks for a tool and your app runs it.
- Regression tests with pytest. An eval case here is a pytest test.
The failure we keep
A support agent was asked: Refund order 1042, it arrived broken. Its order lookup returned an Access denied page (HTTP 403) that the tool wrapper marked ok. With no order data, the agent guessed 49.00 and answered: Done. Refunded 49.00 for order 1042.
Episode 1 fixed the tool wrapper. A fix in one place does not protect you when someone later changes the prompt, the model or the wrapper. An eval case tests the whole agent, so it fails if the bug comes back from any of those.
The agent below is plain Python that stands in for a model and its tool loop, so every run gives the same result. It is not a live model. The order data is made up, and there is no customer data.
What an eval case is
Anthropic's article on evals for AI agents uses these words. We use them the same way.
- Task
- One test with defined inputs and a way to tell success. Ours: refund order 1042 when the order lookup is blocked.
- Grader
- Code that scores the result. Ours is one assert.
- Transcript
- The full record of a run. Also called a trace. Episode 1 read one.
- Outcome
- The final state of the environment. A booking agent can say your flight is booked, but the outcome is whether the reservation exists in the database.
- Trial
- One attempt at a task. Real models vary between runs, so you repeat a task. Our stand-in agent never varies, so we run each case once.
In our case the outcome is the list of refunds the shop recorded. We grade that list. We do not grade the agent's words.
Worked example
Turn the refund failure into a saved case, fix the agent, and make the cases a gate.
Pick the failure to keep
Take the run from episode 1: the order lookup was blocked, and the agent refunded a guessed 49.00. A failure from real work is the best first case. Anthropic advises turning user-reported failures into test cases, and says 20 to 50 simple tasks drawn from real failures is a good start.
Write the case as data
A case holds the task, what the order lookup returns and the outcome you expect. For the blocked run the lookup returns a 403 page, and the expected refunds are an empty list: nothing should be refunded. Put it in cases.json. Data is easy to review, and anyone can add a case without touching code.
Add similar cases
Ask what else looks the same from the agent's side. A maintenance page that comes back with status 200 is a sibling, so it gets a case. Keep one case where everything works (order-ok), so a fix cannot break the normal path.
Grade the outcome
The grader is one assert: after the run, the refunds the shop recorded must equal the refunds the case expects. The answer text is attached only to help you read a failure. If the grader read the words instead, the failed run would pass, because the words say Done.
Run the cases on the current agent
Two of the four tests fail: blocked-403 and maintenance-page. order-ok passes. A suite with only the happy path would have passed, and the bug would have shipped.
Fix the agent and run again
The agent now refunds only when the status is 200 and the body has a total. Otherwise it hands the order to a person. All four tests pass.
Make it a gate
Run the same command in CI. pytest exits with code 1 when a test fails, and CI services such as GitHub Actions treat a non-zero exit as a failed job. Mark that job as a required check, so a branch cannot merge until it passes. The next section covers the empty-suite trap.
Repeat for every new failure
Each new failure gets a case first, then a fix. The suite grows from real failures, and each fix stays fixed.
Run it yourself
Put these files in one folder. Every command here was run on 9 November 2026 with Python 3.12.16 and pytest 9.2.0. Pytest header lines are left out, and timings will differ.
The commands use uv. Without uv, install pytest in a virtual environment and run python -m pytest -v --tb=no -rN instead.
shop.py is a fake order system that records every refund. cases.json holds the three cases. Each case lists the lookup response the shop will give and the refunds you expect.
class Shop:
"""A fake order system. It records every refund, so a case can check what really changed."""
def __init__(self, status, body):
self.status, self.body, self.refunds = status, body, []
def lookup_order(self, order_id):
return self.status, self.body
def issue_refund(self, order_id, amount):
self.refunds.append(amount)[
{
"id": "order-ok",
"task": "Refund order 1042, it arrived broken.",
"lookup": {"status": 200, "body": "{\"order_id\": 1042, \"total\": 52.5}"},
"refunds": [52.5]
},
{
"id": "blocked-403",
"task": "Refund order 1042, it arrived broken.",
"lookup": {"status": 403, "body": "<html><h1>Access denied</h1><p>HTTP 403</p></html>"},
"refunds": []
},
{
"id": "maintenance-page",
"task": "Refund order 1042, it arrived broken.",
"lookup": {"status": 200, "body": "<html>Maintenance</html>"},
"refunds": []
}
]Before the fix, agent.py guesses 49.00 when it cannot read a total. test_evals.py runs every case through the agent and grades the shop's refunds. The first test fails if cases.json is empty.
import json
def total_from(body):
try:
return json.loads(body)["total"]
except (json.JSONDecodeError, KeyError, TypeError):
return None
def agent(task, shop):
"""Stands in for a model and its tool loop, so every run gives the same result."""
status, body = shop.lookup_order(1042)
amount = total_from(body)
if amount is None:
amount = 49.0 # the guess from the failed run
shop.issue_refund(1042, amount)
return f"Done. Refunded {amount:.2f} for order 1042."import json
from pathlib import Path
import pytest
from agent import agent
from shop import Shop
CASES = json.loads(Path(__file__).with_name("cases.json").read_text())
def test_suite_is_not_empty():
assert CASES
@pytest.mark.parametrize("case", CASES, ids=[case["id"] for case in CASES])
def test_case(case):
shop = Shop(case["lookup"]["status"], case["lookup"]["body"])
answer = agent(case["task"], shop)
assert shop.refunds == case["refunds"], answeruv run --no-project --python 3.12 --with pytest python -m pytest -v --tb=no -rNtest_evals.py::test_suite_is_not_empty PASSED [ 25%]
test_evals.py::test_case[order-ok] PASSED [ 50%]
test_evals.py::test_case[blocked-403] FAILED [ 75%]
test_evals.py::test_case[maintenance-page] FAILED [100%]
========================= 2 failed, 2 passed in 0.03s =========================Read one failure. The message is the agent's answer, and the assert line shows the shop holds a refund that the case says should not exist.
uv run --no-project --python 3.12 --with pytest python -m pytest -k blocked --tb=short -qF [100%]
================================== FAILURES ===================================
___________________________ test_case[blocked-403] ____________________________
test_evals.py:20: in test_case
assert shop.refunds == case["refunds"], answer
E AssertionError: Done. Refunded 49.00 for order 1042.
E assert [49.0] == []
E
E Left contains one more item: 49.0
E Use -v to get more diff
=========================== short test summary info ===========================
FAILED test_evals.py::test_case[blocked-403] - AssertionError: Done. Refunded...
1 failed, 3 deselected in 0.12sAfter the fix, agent.py refunds only when the status is 200 and the body has a total. Run the same tests again.
import json
def total_from(body):
try:
return json.loads(body)["total"]
except (json.JSONDecodeError, KeyError, TypeError):
return None
def agent(task, shop):
"""Stands in for a model and its tool loop, so every run gives the same result."""
status, body = shop.lookup_order(1042)
amount = total_from(body) if status == 200 else None
if amount is None:
return "I could not read order 1042, so I did not refund it. A person will check."
shop.issue_refund(1042, amount)
return f"Done. Refunded {amount:.2f} for order 1042."uv run --no-project --python 3.12 --with pytest python -m pytest -v --tb=no -rNtest_evals.py::test_suite_is_not_empty PASSED [ 25%]
test_evals.py::test_case[order-ok] PASSED [ 50%]
test_evals.py::test_case[blocked-403] PASSED [ 75%]
test_evals.py::test_case[maintenance-page] PASSED [100%]
============================== 4 passed in 0.02s ==============================We also read the exit code. pytest exited with 1 on the first run and 0 on the last one, as PowerShell's $LASTEXITCODE showed.
Make the run a release gate
CI reads the exit code. The pytest docs say 0 means all tests were collected and passed, 1 means some failed and 5 means no tests were collected. GitHub's docs on exit codes say a non-zero exit status sets a failure status.
In GitHub you can require a check before a branch merges. Its docs on protected branches say a required check must be successful, skipped or neutral. A skipped check counts as passing, so run the eval job on every pull request. We did not run a CI service for this guide. The tested part is the pytest exit code. The branch setting is as documented.
A gate that can pass with zero cases is not a gate. When a parametrized list is empty, pytest does not fail. The default is to skip the test. That is not exit code 5, because the test is still collected. We emptied cases.json to check. Without test_suite_is_not_empty, pytest printed 1 skipped and exited 0. With it, pytest printed 1 failed, 1 skipped and exited 1.
What this does not do
- The agent here is plain Python, not a model. A real model varies between runs. Anthropic repeats each task in several trials for this reason. With a real model, run each case several times and decide how many failures you accept.
- Anthropic says a regression suite should pass at a rate near 100 percent, because a drop means something broke. That is their guidance, not a rule of pytest.
- Three cases are a start, not coverage. Add one for every new failure.
- The grader checks one outcome, the refunds. It does not check whether the message to the customer is correct or kind. Model graders and human review cover that.
- A case replays a recorded response. If the real service changes its error page, the case will not notice. Refresh cases from real runs, and remove private data first.
Next: is the fix real or luck?
A real model can pass or fail the same case on different runs, so a higher pass count after a fix can be luck. Episode 3 runs an exact test on two pass counts and plans how many runs you need. Read Did the fix work, or was it luck? Compare two versions of an LLM agent.
Failing cases are not only for agents. If a RAG bot gives a wrong answer, save the question and the chunk that should have answered it. Then you can tell a retrieval miss from a generation miss. Read Your RAG bot gave a wrong answer. Was it retrieval or generation? Find out in Python.
A release gate can also guard a model swap. After you shadow-run a cheaper model, write the rule that says when it ships, and run it as a check. Read Your LLM labels each ticket for 2.2 cents. Can a small model take over? Shadow-run it in Python first.
A release gate can also guard the install. After the dependencies are installed, scan the environment for startup hooks and fail the build on a line that is not on an allow list. Read Did a hijacked PyPI package run on your machine? Audit Python startup hooks and pick a cooldown. And at runtime, keep raw API keys out of the agent process by swapping a short-lived ticket at a local broker: read Your coding agent can read your API keys. Give it a ticket instead. Build the broker in Python.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type LLM agent evals and regression tests into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning. Missed episode 1? Read how to find the first bad step in a trace.
Sources
All sources were checked on 9 November 2026. Read each page for the full context.
- Anthropic: Demystifying evals for AI agents
Defines task, trial, grader, transcript and outcome. The outcome is the final state in the environment. Regression evals should have a nearly 100% pass rate. 20 to 50 simple tasks drawn from real failures is a good start, and user-reported failures can become test cases. The old /engineering/ address redirects here.
- pytest: Exit codes
Exit code 0 means all tests were collected and passed. 1 means some tests failed. 5 means no tests were collected.
- pytest: How to parametrize fixtures and test functions
One test function can run over several sets of arguments. If the list is empty, the empty_parameter_set_mark option decides what happens.
- pytest: empty_parameter_set_mark
The action for an empty parameter set. The default is skip.
- GitHub Docs: Setting exit codes for actions
Exit status 0 sets a success status. Any non-zero value sets a failure status.
- GitHub Docs: About protected branches
Required status checks must be successful, skipped or neutral before changes can be merged into a protected branch.