Interactive demoNo signup

Raw trace in. Root cause out.

Pick a failed run from Rootline's sample project. Inspect its steps, or the raw JSON your tracing tool gives you, next to Rootline's diagnosis: the failure, the root cause, the evidence and a suggested fix.

Sample project · not real customer data1,411 synthetic runs analysed by the same code that runs on your traces. Rule-based explanations, plus one AI-written example.
sample-project / runs / failedSynthetic sample data1,411 runs260 failed9 issues

Run: Silent API failure

Trace

Example: Silent API failure. A CRM call failed with a 401 and the bot told the user the booking worked.

sample-0083lead-capture2026-10-17 21:04 UTC
user
I'd like a viewing on Saturday
agent
Thanks! Your viewing is booked.
  1. input
    …prompt…
    output
    call create_lead
    timing
    +0.00s · 1.83s · $0.004
  2. status
    error · HTTP 401 Unauthorized: token expired
    input
    {"name": "Sara", "phone": "+974..."}
    output
    (none)
    timing
    +1.97s · 0.16s
  3. input
    …prompt…
    output
    final
    timing
    +2.16s · 3.08s · $0.004

Silent API failure. A CRM call failed with a 401 and the bot told the user the booking worked. Select a step to see its input and output.

Example traceRaw trace · Langfuse-style JSON · 59 lines
{  "id": "sample-0083",  "name": "lead-capture",  "userId": "atlas-property",  "tags": [    "sales"  ],  "timestamp": "2026-10-17T21:04:00+00:00",  "input": "I'd like a viewing on Saturday",  "output": "Thanks! Your viewing is booked.",  "observations": [    {      "id": "obs-63825046",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "call create_lead",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-17T21:04:00.050000+00:00",      "endTime": "2026-10-17T21:04:01.877565+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    },    {      "id": "obs-99ea4514",      "type": "SPAN",      "name": "crm.create_lead",      "input": {        "name": "Sara",        "phone": "+974..."      },      "output": null,      "level": "ERROR",      "statusMessage": "HTTP 401 Unauthorized: token expired",      "startTime": "2026-10-17T21:04:02.019608+00:00",      "endTime": "2026-10-17T21:04:02.180138+00:00",      "calculatedTotalCost": 0.0,      "model": null    },    {      "id": "obs-894e9f37",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "final",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-17T21:04:02.206236+00:00",      "endTime": "2026-10-17T21:04:05.291047+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    }  ],  "latency": 5.365,  "metadata": {    "rootline_sample": true  }}

Sample data. The model name is a placeholder. Highlighted lines are the failing step.

Rootline diagnosis

Failure

Error raisedhigh

Rule-based Built-in explanation, no AI involved.

Root cause

The run failed because crm.create_lead rejected the agent's credentials (HTTP 401 Unauthorized: token expired). Worse, the agent then told the user "Thanks! Your viewing is booked." even though the call failed: a false success, so the user thinks it worked and nobody follows up.

Suggested fix

Refresh or rotate the token for crm.create_lead (use the refresh-token flow, or renew it before it expires) and re-send the request with the new token.

View evidenceStep 02 · crm.create_lead
Evidence

Step 02 · crm.create_lead

status
HTTP 401 Unauthorized: token expired
in
{"name": "Sara", "phone": "+974..."}
out
(none)
user
I'd like a viewing on Saturday
agent
Thanks! Your viewing is booked.
Show full explanation and fixrule-based text
Root cause

The run failed because crm.create_lead rejected the agent's credentials (HTTP 401 Unauthorized: token expired). Worse, the agent then told the user "Thanks! Your viewing is booked." even though the call failed: a false success, so the user thinks it worked and nobody follows up.

Suggested fix

Refresh or rotate the token for crm.create_lead (use the refresh-token flow, or renew it before it expires) and re-send the request with the new token. Don't retry with the same token: a 401 is not transient. If the refresh fails, return a clear error to the model so it tells the user the action didn't go through. Never let the agent confirm an action unless the tool call actually succeeded.

The summary above is the opening of this text; this is the full original.

Grouped into issue

Error raised in crm.create_lead (lead-capture)

34 runs$0.27customers: Team and Agency

View regression testpytest · rootline_error_1
Generated regression test

Test rootline_error_1 replays this run's recorded tool outputs as mocks, so it is deterministic, and fails if the agent repeats the bug: the recorded tool error is replayed, and the answer must not claim the action succeeded.

"""Regression tests generated by Rootline from real failed runs.Each case replays the tool outputs recorded in the failing run, so the test is deterministic and needs no live APIs.Wire `run_agent(user_input, tools)` to your agent: route every tool call through `tools(name, input)` andreturn dict(output=str, latency_s=float, cost=float)."""import json, re, pytestCASES = json.loads(r"""[  {    "id": "rootline_error_1",    "issue": "Error raised in crm.create_lead (lead-capture)",    "source_run": "sample-0083",    "feature": "lead-capture",    "input": "I'd like a viewing on Saturday",    "bad_output": "Thanks! Your viewing is booked.",    "kind": "error",    "tool_outputs": [      {        "name": "crm.create_lead",        "input": {          "name": "Sara",          "phone": "+974..."        },        "output": null,        "level": "ERROR",        "error": "HTTP 401 Unauthorized: token expired"      }    ],    "assertions": [      "the recorded tool error is replayed, and the answer must not claim the action succeeded"    ]  }]""")class ToolError(Exception):    passclass Replay:    """Mock tools: returns the outputs recorded in the failing run, in order per tool name (repeating the last one)."""    def __init__(self, recorded):        self.recorded, self.calls, self._pos = recorded, [], {}    def __call__(self, name, tool_input=""):        self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str)))        mine = [t for t in self.recorded if t["name"] == name]        if not mine:            raise ToolError(f"tool {name!r} was not called in the recorded run")        i = self._pos.get(name, 0)        self._pos[name] = i + 1        t = mine[min(i, len(mine) - 1)]        if t.get("error") and "timed out" in t["error"].lower():            raise TimeoutError(t["error"])        if t.get("level") == "ERROR":            raise ToolError(t.get("error") or "tool error")        return t["output"]def run_agent(user_input, tools):    raise NotImplementedError("Connect this to your agent entrypoint and pass `tools` in place of your real tool calls")UNCERTAIN = ("don't know", "do not know", "couldn't find", "could not find", "not sure", "no information", "unable to find",             "can't find", "cannot find", "?")SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|is done|all set|been (sent|created|saved|submitted|placed)|successfully|went through)\b", re.I)NUM = re.compile(r"\d[\d,]*(?:\.\d+)?")@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    res = run_agent(case["input"], tools)    out = res["output"]    k = case["kind"]    if k in ("error", "timeout"):        assert not SUCCESS.search(out), "agent claimed success after the tool failed"        if k == "timeout":            assert res.get("latency_s", 0) <= 60    elif k == "loop":        counts = {}        for c in tools.calls:            counts[c] = counts.get(c, 0) + 1        assert max(counts.values(), default=0) < 3    elif k == "high_cost":        assert res.get("cost", 0) <= 0.50    elif k == "empty_tool_output":        assert any(u in out.lower() for u in UNCERTAIN), "answer should admit it found nothing or ask a question"    elif k == "tool_output_ignored":        facts = case["expect_facts"]        hit = [f for f in facts if f.lower() in out.lower()]        assert len(hit) >= max(1, (len(facts) + 1) // 2), f"answer ignores the tool result; expected most of {facts}"    elif k == "malformed_tool_output":        seen = json.dumps(case["input"], default=str) + json.dumps([t["output"] for t in case["tool_outputs"]], default=str)        known = {n.replace(",", "") for n in NUM.findall(seen)}        made_up = [n for n in NUM.findall(out) if n.replace(",", "") not in known]        assert not made_up, f"answer states figures not in the input or any tool output: {made_up}"

The real pytest export format. Downloading JSON or pytest files is a Team and Agency feature; Free can view generated cases in the dashboard.

Synthetic sample run

Walkthrough: a tool returns an error page, the agent promises a refund

One run from the sample project, step by step, exactly as Rootline processes it. Every value below is read from the same data as the workspace above.

  1. The run. A customer writes: My blender arrived broken. Can I get a refund and a replacement? The agent (refund-agent) calls get_order, search_policy, refunds_api.check_eligibility across 7 steps.
  2. Tools return unexpected results. Step 04 search_policy returns {"hits": []}. Step 06 refunds_api.check_eligibility returns a 503 HTML error page instead of JSON.
  3. The agent decides wrongly. Its final answer: Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow. No tool output contains that amount or date.
  4. Inspect the trace. In the timeline the failing step is marked; in Raw JSON its lines are highlighted. Open this run in the workspace.
  5. Rootline identifies the failure. Malformed tool output (medium), also detected: Empty tool output. It is grouped into the issue Malformed tool output in refunds_api.check_eligibility (refund-agent), 6 runs in this sample project.
  6. Recommended correction. Treat a non-JSON or 5xx response from refunds_api.check_eligibility as a failure: retry once, then tell the customer you'll confirm by email and open a ticket for a human.
  7. Regression test. Rootline generates rootline_malformed_tool_output_1, which replays the recorded tool outputs as mocks. What it checks: the recorded broken output is replayed, and the answer states no figure that isn't in the user input or tool outputs. Downloading the pytest or JSON export is a Team and Agency feature; Free can view generated cases.

Reproduce it in your own account: sign up free, then in Settings choose Use sample project. The same synthetic runs are analysed by the same detectors. In your account the explanation is generated fresh (AI-written when AI explanations are on, rule-based otherwise), so its wording will differ from the pre-generated text shown here.

Grouped into issues

260 failed runs out of 1411 in this sample project fall into 9 issues; the 5 most severe are listed. Rootline groups by failure type, feature and tool. Which customers an issue affects is a Team and Agency feature.

Sample dataIssues · ranked by severity
IssueSeverityRunsCost
Agent loop in check_availability (booking-agent)critical32$13.21
Agent loop in create_shipment (order-agent)critical5$0.48
Empty tool output in search_kb (support-bot)high62$0.50
Tool output ignored in get_order (order-agent)high50$0.40
Error raised in crm.create_lead (lead-capture)high34$0.27

Sample dataTeam and Agency feature

Cost per customer, and a runaway-loop alert

Total cost across the 1411 sample runs: $39.65. Team and Agency break it down by customer, feature and bot or API key. Free shows overall totals and daily spend only.

Top customers by cost
CustomerRunsCost
harbor-insurance117$4.92
northwind-legal122$4.48
brightpath-realty116$3.99
acme-dental123$3.86
zen-fitness130$3.64
Most expensive single runs
CustomerFeatureRun cost
harbor-insurancereport-writer$1.00
qatar-motors-demoreport-writer$0.94
northwind-legalreport-writer$0.93
pinecrest-hotelsreport-writer$0.93
acme-dentalreport-writer$0.82
critical
Runaway loop: run sample-0184 (booking-agent, user cobalt-logistics) cost $0.24
Alert example · agent loop. The booking-agent loop issue in this sample project covers 32 runs and $13.21. Team and Agency get an email when an agent loops or spend spikes.

What you just saw, and what is different on your data

Sample project

Everything on this page comes from 1411 synthetic runs in Rootline's built-in sample project. It is not real customer data. The analysis is the same code that runs on your traces.

Explanations

Most examples here show Rootline's built-in rule-based explanations. The messy refund example shows an AI-written explanation, generated ahead of time for this sample: this page never calls an AI provider. After you sign up, your failed runs get AI-written explanations from a third-party AI provider. You can turn that off in your workspace settings, and then nothing from your traces is sent.

Your traces

Connect Langfuse with your project keys (no code changes), or point an OpenTelemetry OTLP/HTTP exporter at Rootline. The Free plan includes AI explanations for 100 failed runs per month.

Run this on your own traces

Connect Langfuse or send OpenTelemetry traces and see your own failed runs explained, grouped and turned into tests.

Start debugging free creates a free account, no card. Then connect Langfuse with your project keys or send OpenTelemetry traces.