Guide

AI agent regression testing from real failed runs

A fixed bug should stay fixed. For agents that means a test built from the run that failed, replaying the same tool results, and checking the behaviour that went wrong.

RootlinePublished

Short answer

Build AI agent regression tests from the runs that actually failed. Save the run's user input and every tool output it received, replay those outputs through mock tools so the test needs no live APIs, run your agent against them, and assert on behaviour: it doesn't claim success after a failed call, doesn't repeat the same call three times, admits when a search came back empty, uses the facts the tool returned. Don't assert on exact wording; model output varies.

Synthetic example · not customer data

A calendar agent told a user "Your meeting with Dana is booked for 3pm" after the create_event tool returned 401 token expired. The team fixed the token refresh and added a rule that the agent must not confirm an action whose tool call failed. They want a test that fails if either fix regresses.

Build a test from a failed run

  1. Capture the run. From the trace, take the user input, the final answer, and each tool call in order: tool name, input, output, and whether it errored.
  2. Make tools injectable. Your agent entry point should accept a tool-calling function, so a test can pass a mock instead of real APIs. This is the one code change most teams need.
  3. Replay recorded outputs. The mock returns what each tool returned in the failed run, in order, and raises when the recorded step was an error or timeout.
  4. Assert the behaviour that went wrong. One assertion per failure type (see below). Keep it about what the agent did, not how it phrased it.
  5. Run it before and after the fix. It should fail on the old code and pass on the new. If it passes on both, it isn't testing the bug.
  6. Run it in CI. Model output is not deterministic, so use a low temperature, and consider running each case a few times before marking it flaky.

A pytest example

import json, re, pytestCASES = [{    "id": "false_success_create_event",    "input": "Book a meeting with Dana tomorrow at 3pm",    "kind": "error",    "tool_outputs": [        {"name": "find_slot", "output": {"start": "15:00", "free": True}, "level": "DEFAULT"},        {"name": "create_event", "output": None, "level": "ERROR", "error": "401 token expired"},    ],}]class ToolError(Exception):    passclass Replay:    """Mock tools: return the recorded outputs in order per tool name."""    def __init__(self, recorded):        self.recorded, self.calls, self.pos = recorded, [], {}    def __call__(self, name, tool_input=""):        self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str)))        mine = [t for t in self.recorded if t["name"] == name]        i = self.pos[name] = self.pos.get(name, -1) + 1        t = mine[min(i, len(mine) - 1)]        if t.get("level") == "ERROR":            raise ToolError(t.get("error") or "tool error")        return t["output"]SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|all set|successfully)\b", re.I)@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    out = run_agent(case["input"], tools)["output"]      # your agent, with tools injected    assert not SUCCESS.search(out), "agent claimed success after the tool failed"

run_agent is your code: call your agent with the user input and route every tool call through tools(name, input).

What to assert, by failure type

FailureAssertion
Error or timeoutThe answer does not claim the action succeeded; for timeouts, the run finishes within your latency budget
Agent loopNo tool is called three or more times with identical input on the replayed tools
Empty tool outputThe answer says it couldn't find the information, or asks a clarifying question
Tool output ignoredThe answer repeats most of the key facts the tool returned
Malformed tool outputThe answer states no number that isn't in the user input or a tool output
High costThe run's cost stays under your budget

Common mistakes

  • Calling live APIs in tests. The bug came from a specific tool response; replay that response, don't hope the API produces it again.
  • Snapshotting the exact answer. It breaks on every prompt tweak and says nothing about the bug.
  • Writing tests from imagined failures instead of recorded ones. Real failed runs are the best test data you have.
  • One giant test. One case per failure pattern makes it obvious which behaviour regressed.
  • Never checking the test fails without the fix.

When Rootline helps

Where it fits

On the Team plan, Rootline's Tests page exports one test case per issue, built from that issue's latest failed run, as JSON or as a ready-to-run pytest file. Each case carries the run's input, the bad output, the failure type, every recorded tool call with its output and error, and the assertion for that failure type (the checks in the table above, with 60 seconds and $0.50 as the limits); "tool output ignored" cases also list the facts the answer is expected to contain. The pytest file includes a Replay mock like the one above. You write run_agent to call your agent with the mock tools.

Limitations

Rootline doesn't run the tests or connect to your CI; it generates the cases and the test file. Your agent must accept injectable tools. When the agent calls a tool more times than were recorded, the mock repeats the last recorded output. Assertions are behavioural checks with simple patterns, so a phrasing the regex doesn't know can slip through. For scored quality gates on curated datasets, Langfuse datasets and experiments can run in CI as well.

Official documentation

Find out why your last failed run broke

Rootline reads your Langfuse or OpenTelemetry traces and explains each failed run: the failing step, the likely cause and a suggested fix. Free to start, no credit card.

Start debugging free creates a free account, no card. Then connect Langfuse or OpenTelemetry.