Interactive demoNo signup

Raw trace in. Root cause out.

Pick a failed run from Rootline's sample project. Inspect its steps, or the raw JSON your tracing tool gives you, next to Rootline's diagnosis: the failure, the root cause, the evidence and a suggested fix.

Sample project · not real customer data1,411 synthetic runs analysed by the same code that runs on your traces. Rule-based explanations, plus one AI-written example.
sample-project / runs / failedSynthetic sample data1,411 runs260 failed9 issues

Run: Messy refund (AI)

Trace

Example: Messy refund (AI). Three steps went wrong in one run, and the bot promised a refund anyway.

sample-1400refund-agent2026-10-21 05:00 UTC
user
My blender arrived broken. Can I get a refund and a replacement?
agent
Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow.
  1. input
    …prompt…
    output
    call get_order
    timing
    +0.00s · 1.12s · $0.004
  2. input
    {"order_id": "B-2207"}
    output
    {"order_id": "B-2207", "status": "delivered", "item": "Blender X200", "delivered_on": "2026-10-12"}
    timing
    +1.30s · 0.75s
  3. input
    …prompt…
    output
    call search_policy
    timing
    +2.24s · 3.09s · $0.004
  4. input
    {"q": "damaged item refund replacement"}
    output
    {"hits": []}
    timing
    +5.43s · 0.25s
  5. input
    …prompt…
    output
    call refunds_api
    timing
    +5.84s · 0.90s · $0.004
  6. input
    {"order_id": "B-2207", "reason": "damaged"}
    output
    <!DOCTYPE html><html><head><title>503 Service Unavailable</title></head><body>upstream connect error</body></html>
    timing
    +6.84s · 1.01s
  7. input
    …prompt…
    output
    final
    timing
    +7.89s · 1.78s · $0.011

Messy refund (AI). Three steps went wrong in one run, and the bot promised a refund anyway. Select a step to see its input and output.

Example traceRaw trace · Langfuse-style JSON · 122 lines
{  "id": "sample-1400",  "name": "refund-agent",  "userId": "harbor-insurance",  "tags": [    "support"  ],  "timestamp": "2026-10-21T05:00:00+00:00",  "input": "My blender arrived broken. Can I get a refund and a replacement?",  "output": "Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow.",  "observations": [    {      "id": "obs-2392726b",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "call get_order",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:00.050000+00:00",      "endTime": "2026-10-21T05:00:01.174545+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    },    {      "id": "obs-91786490",      "type": "SPAN",      "name": "get_order",      "input": {        "order_id": "B-2207"      },      "output": {        "order_id": "B-2207",        "status": "delivered",        "item": "Blender X200",        "delivered_on": "2026-10-12"      },      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:01.351086+00:00",      "endTime": "2026-10-21T05:00:02.104737+00:00",      "calculatedTotalCost": 0.0,      "model": null    },    {      "id": "obs-2d96cbd9",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "call search_policy",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:02.285463+00:00",      "endTime": "2026-10-21T05:00:05.376485+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    },    {      "id": "obs-8b5166de",      "type": "SPAN",      "name": "search_policy",      "input": {        "q": "damaged item refund replacement"      },      "output": {        "hits": []      },      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:05.479125+00:00",      "endTime": "2026-10-21T05:00:05.732230+00:00",      "calculatedTotalCost": 0.0,      "model": null    },    {      "id": "obs-7cae3e05",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "call refunds_api",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:05.890639+00:00",      "endTime": "2026-10-21T05:00:06.794024+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    },    {      "id": "obs-13d30757",      "type": "SPAN",      "name": "refunds_api.check_eligibility",      "input": {        "order_id": "B-2207",        "reason": "damaged"      },      "output": "<!DOCTYPE html><html><head><title>503 Service Unavailable</title></head><body>upstream connect error</body></html>",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:06.891829+00:00",      "endTime": "2026-10-21T05:00:07.900431+00:00",      "calculatedTotalCost": 0.0,      "model": null    },    {      "id": "obs-604d9d27",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "final",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:00:07.935932+00:00",      "endTime": "2026-10-21T05:00:09.710951+00:00",      "calculatedTotalCost": 0.011,      "model": "example-model"    }  ],  "latency": 9.821,  "metadata": {    "rootline_sample": true  }}

Sample data. The model name is a placeholder. Highlighted lines are the failing step.

Rootline diagnosis

Failure

Malformed tool outputmedium

AI-written Pre-generated for this sample. This page never calls an AI provider.

Root cause

The policy search came back empty and refunds_api.check_eligibility returned a 503 HTML error page, yet the agent promised a full $349 refund and a replacement "tomorrow" that no tool output supports.

Also detectedEmpty tool output

Suggested fix

Treat a non-JSON or 5xx response from refunds_api.check_eligibility as a failure: retry once, then tell the customer you'll confirm by email and open a ticket for a human.

View evidenceStep 06 · refunds_api.check_eligibility
Evidence

Step 06 · refunds_api.check_eligibility

in
{"order_id": "B-2207", "reason": "damaged"}
out
<!DOCTYPE html><html><head><title>503 Service Unavailable</title></head><body>upstream connect error</body></html>
user
My blender arrived broken. Can I get a refund and a replacement?
agent
Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow.
Show full explanation and fixAI-written text
Root cause

Three things went wrong in a row, and the agent papered over all of them. The policy search for damaged items came back empty, then refunds_api.check_eligibility returned a 503 HTML error page instead of an eligibility decision. With no policy and no eligibility answer, the agent still promised a full $349 refund and a replacement "tomorrow". Neither the amount nor the date appears in any tool output, so the customer now expects money and a shipment nobody approved.

Suggested fix

Treat a non-JSON or 5xx response from refunds_api.check_eligibility as a failure: retry once, then tell the customer you'll confirm by email and open a ticket for a human. Don't let the model state amounts or dates that aren't in a tool result. Separately, find out why search_policy returns nothing for damaged items: the damaged-goods policy is probably missing from the index.

The summary above condenses this AI-written text; this is the original.

Grouped into issue

Malformed tool output in refunds_api.check_eligibility (refund-agent)

6 runs$0.14customers: Team and Agency

View regression testpytest · rootline_malformed_tool_output_1
Generated regression test

Test rootline_malformed_tool_output_1 replays this run's recorded tool outputs as mocks, so it is deterministic, and fails if the agent repeats the bug: the recorded broken output is replayed, and the answer states no figure that isn't in the user input or tool outputs.

"""Regression tests generated by Rootline from real failed runs.Each case replays the tool outputs recorded in the failing run, so the test is deterministic and needs no live APIs.Wire `run_agent(user_input, tools)` to your agent: route every tool call through `tools(name, input)` andreturn dict(output=str, latency_s=float, cost=float)."""import json, re, pytestCASES = json.loads(r"""[  {    "id": "rootline_malformed_tool_output_1",    "issue": "Malformed tool output in refunds_api.check_eligibility (refund-agent)",    "source_run": "sample-1400",    "feature": "refund-agent",    "input": "My blender arrived broken. Can I get a refund and a replacement?",    "bad_output": "Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow.",    "kind": "malformed_tool_output",    "tool_outputs": [      {        "name": "get_order",        "input": {          "order_id": "B-2207"        },        "output": {          "order_id": "B-2207",          "status": "delivered",          "item": "Blender X200",          "delivered_on": "2026-10-12"        },        "level": "DEFAULT",        "error": null      },      {        "name": "search_policy",        "input": {          "q": "damaged item refund replacement"        },        "output": {          "hits": []        },        "level": "DEFAULT",        "error": null      },      {        "name": "refunds_api.check_eligibility",        "input": {          "order_id": "B-2207",          "reason": "damaged"        },        "output": "<!DOCTYPE html><html><head><title>503 Service Unavailable</title></head><body>upstream connect error</body></html>",        "level": "DEFAULT",        "error": null      }    ],    "assertions": [      "the recorded broken output is replayed, and the answer states no figure that isn't in the user input or tool outputs"    ]  }]""")class ToolError(Exception):    passclass Replay:    """Mock tools: returns the outputs recorded in the failing run, in order per tool name (repeating the last one)."""    def __init__(self, recorded):        self.recorded, self.calls, self._pos = recorded, [], {}    def __call__(self, name, tool_input=""):        self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str)))        mine = [t for t in self.recorded if t["name"] == name]        if not mine:            raise ToolError(f"tool {name!r} was not called in the recorded run")        i = self._pos.get(name, 0)        self._pos[name] = i + 1        t = mine[min(i, len(mine) - 1)]        if t.get("error") and "timed out" in t["error"].lower():            raise TimeoutError(t["error"])        if t.get("level") == "ERROR":            raise ToolError(t.get("error") or "tool error")        return t["output"]def run_agent(user_input, tools):    raise NotImplementedError("Connect this to your agent entrypoint and pass `tools` in place of your real tool calls")UNCERTAIN = ("don't know", "do not know", "couldn't find", "could not find", "not sure", "no information", "unable to find",             "can't find", "cannot find", "?")SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|is done|all set|been (sent|created|saved|submitted|placed)|successfully|went through)\b", re.I)NUM = re.compile(r"\d[\d,]*(?:\.\d+)?")@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    res = run_agent(case["input"], tools)    out = res["output"]    k = case["kind"]    if k in ("error", "timeout"):        assert not SUCCESS.search(out), "agent claimed success after the tool failed"        if k == "timeout":            assert res.get("latency_s", 0) <= 60    elif k == "loop":        counts = {}        for c in tools.calls:            counts[c] = counts.get(c, 0) + 1        assert max(counts.values(), default=0) < 3    elif k == "high_cost":        assert res.get("cost", 0) <= 0.50    elif k == "empty_tool_output":        assert any(u in out.lower() for u in UNCERTAIN), "answer should admit it found nothing or ask a question"    elif k == "tool_output_ignored":        facts = case["expect_facts"]        hit = [f for f in facts if f.lower() in out.lower()]        assert len(hit) >= max(1, (len(facts) + 1) // 2), f"answer ignores the tool result; expected most of {facts}"    elif k == "malformed_tool_output":        seen = json.dumps(case["input"], default=str) + json.dumps([t["output"] for t in case["tool_outputs"]], default=str)        known = {n.replace(",", "") for n in NUM.findall(seen)}        made_up = [n for n in NUM.findall(out) if n.replace(",", "") not in known]        assert not made_up, f"answer states figures not in the input or any tool output: {made_up}"

The real pytest export format. Downloading JSON or pytest files is a Team and Agency feature; Free can view generated cases in the dashboard.

Synthetic sample run

Walkthrough: a tool returns an error page, the agent promises a refund

One run from the sample project, step by step, exactly as Rootline processes it. Every value below is read from the same data as the workspace above.

  1. The run. A customer writes: My blender arrived broken. Can I get a refund and a replacement? The agent (refund-agent) calls get_order, search_policy, refunds_api.check_eligibility across 7 steps.
  2. Tools return unexpected results. Step 04 search_policy returns {"hits": []}. Step 06 refunds_api.check_eligibility returns a 503 HTML error page instead of JSON.
  3. The agent decides wrongly. Its final answer: Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow. No tool output contains that amount or date.
  4. Inspect the trace. In the timeline the failing step is marked; in Raw JSON its lines are highlighted. Open this run in the workspace.
  5. Rootline identifies the failure. Malformed tool output (medium), also detected: Empty tool output. It is grouped into the issue Malformed tool output in refunds_api.check_eligibility (refund-agent), 6 runs in this sample project.
  6. Recommended correction. Treat a non-JSON or 5xx response from refunds_api.check_eligibility as a failure: retry once, then tell the customer you'll confirm by email and open a ticket for a human.
  7. Regression test. Rootline generates rootline_malformed_tool_output_1, which replays the recorded tool outputs as mocks. What it checks: the recorded broken output is replayed, and the answer states no figure that isn't in the user input or tool outputs. Downloading the pytest or JSON export is a Team and Agency feature; Free can view generated cases.

Reproduce it in your own account: sign up free, then in Settings choose Use sample project. The same synthetic runs are analysed by the same detectors. In your account the explanation is generated fresh (AI-written when AI explanations are on, rule-based otherwise), so its wording will differ from the pre-generated text shown here.

Grouped into issues

260 failed runs out of 1411 in this sample project fall into 9 issues; the 5 most severe are listed. Rootline groups by failure type, feature and tool. Which customers an issue affects is a Team and Agency feature.

Sample dataIssues · ranked by severity
IssueSeverityRunsCost
Agent loop in check_availability (booking-agent)critical32$13.21
Agent loop in create_shipment (order-agent)critical5$0.48
Empty tool output in search_kb (support-bot)high62$0.50
Tool output ignored in get_order (order-agent)high50$0.40
Error raised in crm.create_lead (lead-capture)high34$0.27

Sample dataTeam and Agency feature

Cost per customer, and a runaway-loop alert

Total cost across the 1411 sample runs: $39.65. Team and Agency break it down by customer, feature and bot or API key. Free shows overall totals and daily spend only.

Top customers by cost
CustomerRunsCost
harbor-insurance117$4.92
northwind-legal122$4.48
brightpath-realty116$3.99
acme-dental123$3.86
zen-fitness130$3.64
Most expensive single runs
CustomerFeatureRun cost
harbor-insurancereport-writer$1.00
qatar-motors-demoreport-writer$0.94
northwind-legalreport-writer$0.93
pinecrest-hotelsreport-writer$0.93
acme-dentalreport-writer$0.82
critical
Runaway loop: run sample-0184 (booking-agent, user cobalt-logistics) cost $0.24
Alert example · agent loop. The booking-agent loop issue in this sample project covers 32 runs and $13.21. Team and Agency get an email when an agent loops or spend spikes.

What you just saw, and what is different on your data

Sample project

Everything on this page comes from 1411 synthetic runs in Rootline's built-in sample project. It is not real customer data. The analysis is the same code that runs on your traces.

Explanations

Most examples here show Rootline's built-in rule-based explanations. The messy refund example shows an AI-written explanation, generated ahead of time for this sample: this page never calls an AI provider. After you sign up, your failed runs get AI-written explanations from a third-party AI provider. You can turn that off in your workspace settings, and then nothing from your traces is sent.

Your traces

Connect Langfuse with your project keys (no code changes), or point an OpenTelemetry OTLP/HTTP exporter at Rootline. The Free plan includes AI explanations for 100 failed runs per month.

Run this on your own traces

Connect Langfuse or send OpenTelemetry traces and see your own failed runs explained, grouped and turned into tests.

Start debugging free creates a free account, no card. Then connect Langfuse with your project keys or send OpenTelemetry traces.