Build AI agent regression tests from the runs that actually failed. Save the run's user input and every tool output it received, replay those outputs through mock tools so the test needs no live APIs, run your agent against them, and assert on behaviour: it doesn't claim success after a failed call, doesn't repeat the same call three times, admits when a search came back empty, uses the facts the tool returned. Don't assert on exact wording; model output varies.
A calendar agent told a user "Your meeting with Dana is booked for 3pm" after the create_event tool returned 401 token expired. The team fixed the token refresh and added a rule that the agent must not confirm an action whose tool call failed. They want a test that fails if either fix regresses.
Build a test from a failed run
- Capture the run. From the trace, take the user input, the final answer, and each tool call in order: tool name, input, output, and whether it errored.
- Make tools injectable. Your agent entry point should accept a tool-calling function, so a test can pass a mock instead of real APIs. This is the one code change most teams need.
- Replay recorded outputs. The mock returns what each tool returned in the failed run, in order, and raises when the recorded step was an error or timeout.
- Assert the behaviour that went wrong. One assertion per failure type (see below). Keep it about what the agent did, not how it phrased it.
- Run it before and after the fix. It should fail on the old code and pass on the new. If it passes on both, it isn't testing the bug.
- Run it in CI. Model output is not deterministic, so use a low temperature, and consider running each case a few times before marking it flaky.
A pytest example
import json, re, pytestCASES = [{ "id": "false_success_create_event", "input": "Book a meeting with Dana tomorrow at 3pm", "kind": "error", "tool_outputs": [ {"name": "find_slot", "output": {"start": "15:00", "free": True}, "level": "DEFAULT"}, {"name": "create_event", "output": None, "level": "ERROR", "error": "401 token expired"}, ],}]class ToolError(Exception): passclass Replay: """Mock tools: return the recorded outputs in order per tool name.""" def __init__(self, recorded): self.recorded, self.calls, self.pos = recorded, [], {} def __call__(self, name, tool_input=""): self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str))) mine = [t for t in self.recorded if t["name"] == name] i = self.pos[name] = self.pos.get(name, -1) + 1 t = mine[min(i, len(mine) - 1)] if t.get("level") == "ERROR": raise ToolError(t.get("error") or "tool error") return t["output"]SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|all set|successfully)\b", re.I)@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case): tools = Replay(case["tool_outputs"]) out = run_agent(case["input"], tools)["output"] # your agent, with tools injected assert not SUCCESS.search(out), "agent claimed success after the tool failed"
run_agent is your code: call your agent with the user input and route every tool call through tools(name, input).
What to assert, by failure type
| Failure | Assertion |
|---|---|
| Error or timeout | The answer does not claim the action succeeded; for timeouts, the run finishes within your latency budget |
| Agent loop | No tool is called three or more times with identical input on the replayed tools |
| Empty tool output | The answer says it couldn't find the information, or asks a clarifying question |
| Tool output ignored | The answer repeats most of the key facts the tool returned |
| Malformed tool output | The answer states no number that isn't in the user input or a tool output |
| High cost | The run's cost stays under your budget |
Common mistakes
- Calling live APIs in tests. The bug came from a specific tool response; replay that response, don't hope the API produces it again.
- Snapshotting the exact answer. It breaks on every prompt tweak and says nothing about the bug.
- Writing tests from imagined failures instead of recorded ones. Real failed runs are the best test data you have.
- One giant test. One case per failure pattern makes it obvious which behaviour regressed.
- Never checking the test fails without the fix.
When Rootline helps
On the Team plan, Rootline's Tests page exports one test case per issue, built from that issue's latest failed run, as JSON or as a ready-to-run pytest file. Each case carries the run's input, the bad output, the failure type, every recorded tool call with its output and error, and the assertion for that failure type (the checks in the table above, with 60 seconds and $0.50 as the limits); "tool output ignored" cases also list the facts the answer is expected to contain. The pytest file includes a Replay mock like the one above. You write run_agent to call your agent with the mock tools.
Rootline doesn't run the tests or connect to your CI; it generates the cases and the test file. Your agent must accept injectable tools. When the agent calls a tool more times than were recorded, the mock repeats the last recorded output. Assertions are behavioural checks with simple patterns, so a phrasing the regex doesn't know can slip through. For scored quality gates on curated datasets, Langfuse datasets and experiments can run in CI as well.
Official documentation
- pytest: parametrizing testsdocs.pytest.org
- pytest: monkeypatching and mockingdocs.pytest.org
- Python unittest.mockdocs.python.org
- Langfuse datasetslangfuse.com
- Langfuse experiments in CI/CDlangfuse.com