Interactive demoNo signup

Raw trace in. Root cause out.

Pick a failed run from Rootline's sample project. Inspect its steps, or the raw JSON your tracing tool gives you, next to Rootline's diagnosis: the failure, the root cause, the evidence and a suggested fix.

Sample project · not real customer data1,411 synthetic runs analysed by the same code that runs on your traces. Rule-based explanations, plus one AI-written example.
sample-project / runs / failedSynthetic sample data1,411 runs260 failed9 issues

Run: Made-up answer

Trace

Example: Made-up answer. The knowledge-base search found nothing and the bot answered anyway.

sample-0007support-bot2026-10-21 05:25 UTC
user
Can I get a refund on my annual plan?
agent
Yes, annual plans can be refunded in full within 60 days.
  1. input
    …prompt…
    output
    call search_kb
    timing
    +0.00s · 1.99s · $0.004
  2. input
    {"q": "refund policy for annual plan"}
    output
    {"hits": []}
    timing
    +2.15s · 0.68s
  3. input
    …prompt…
    output
    final
    timing
    +2.87s · 2.83s · $0.004

Made-up answer. The knowledge-base search found nothing and the bot answered anyway. Select a step to see its input and output.

Example traceRaw trace · Langfuse-style JSON · 60 lines
{  "id": "sample-0007",  "name": "support-bot",  "userId": "qatar-motors-demo",  "tags": [    "support"  ],  "timestamp": "2026-10-21T05:25:00+00:00",  "input": "Can I get a refund on my annual plan?",  "output": "Yes, annual plans can be refunded in full within 60 days.",  "observations": [    {      "id": "obs-2eae05cf",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "call search_kb",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:25:00.050000+00:00",      "endTime": "2026-10-21T05:25:02.039937+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    },    {      "id": "obs-482c9cbc",      "type": "SPAN",      "name": "search_kb",      "input": {        "q": "refund policy for annual plan"      },      "output": {        "hits": []      },      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:25:02.200193+00:00",      "endTime": "2026-10-21T05:25:02.875902+00:00",      "calculatedTotalCost": 0.0,      "model": null    },    {      "id": "obs-254b0c4e",      "type": "GENERATION",      "name": "llm",      "input": "\u2026prompt\u2026",      "output": "final",      "level": "DEFAULT",      "statusMessage": null,      "startTime": "2026-10-21T05:25:02.924766+00:00",      "endTime": "2026-10-21T05:25:05.758691+00:00",      "calculatedTotalCost": 0.004,      "model": "example-model"    }  ],  "latency": 5.997,  "metadata": {    "rootline_sample": true  }}

Sample data. The model name is a placeholder. Highlighted lines are the failing step.

Rootline diagnosis

Failure

Empty tool outputhigh

Rule-based Built-in explanation, no AI involved.

Root cause

search_kb returned an empty result, and the agent answered anyway instead of saying it didn't know, so the answer is likely made up.

Suggested fix

When search_kb returns nothing, have the agent say it couldn't find the information (or ask a clarifying question) instead of answering from guesswork.

View evidenceStep 02 · search_kb
Evidence

Step 02 · search_kb

in
{"q": "refund policy for annual plan"}
out
{"hits": []}
user
Can I get a refund on my annual plan?
agent
Yes, annual plans can be refunded in full within 60 days.
Show full explanation and fixrule-based text
Root cause

search_kb returned an empty result, and the agent answered anyway instead of saying it didn't know, so the answer is likely made up.

Suggested fix

When search_kb returns nothing, have the agent say it couldn't find the information (or ask a clarifying question) instead of answering from guesswork. Also check the query/index behind search_kb.

The summary above is the opening of this text; this is the full original.

Grouped into issue

Empty tool output in search_kb (support-bot)

62 runs$0.50customers: Team and Agency

View regression testpytest · rootline_empty_tool_output_1
Generated regression test

Test rootline_empty_tool_output_1 replays this run's recorded tool outputs as mocks, so it is deterministic, and fails if the agent repeats the bug: the recorded empty result is replayed, and the answer must say it couldn't find the information or ask a question.

"""Regression tests generated by Rootline from real failed runs.Each case replays the tool outputs recorded in the failing run, so the test is deterministic and needs no live APIs.Wire `run_agent(user_input, tools)` to your agent: route every tool call through `tools(name, input)` andreturn dict(output=str, latency_s=float, cost=float)."""import json, re, pytestCASES = json.loads(r"""[  {    "id": "rootline_empty_tool_output_1",    "issue": "Empty tool output in search_kb (support-bot)",    "source_run": "sample-0007",    "feature": "support-bot",    "input": "Can I get a refund on my annual plan?",    "bad_output": "Yes, annual plans can be refunded in full within 60 days.",    "kind": "empty_tool_output",    "tool_outputs": [      {        "name": "search_kb",        "input": {          "q": "refund policy for annual plan"        },        "output": {          "hits": []        },        "level": "DEFAULT",        "error": null      }    ],    "assertions": [      "the recorded empty result is replayed, and the answer must say it couldn't find the information or ask a question"    ]  }]""")class ToolError(Exception):    passclass Replay:    """Mock tools: returns the outputs recorded in the failing run, in order per tool name (repeating the last one)."""    def __init__(self, recorded):        self.recorded, self.calls, self._pos = recorded, [], {}    def __call__(self, name, tool_input=""):        self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str)))        mine = [t for t in self.recorded if t["name"] == name]        if not mine:            raise ToolError(f"tool {name!r} was not called in the recorded run")        i = self._pos.get(name, 0)        self._pos[name] = i + 1        t = mine[min(i, len(mine) - 1)]        if t.get("error") and "timed out" in t["error"].lower():            raise TimeoutError(t["error"])        if t.get("level") == "ERROR":            raise ToolError(t.get("error") or "tool error")        return t["output"]def run_agent(user_input, tools):    raise NotImplementedError("Connect this to your agent entrypoint and pass `tools` in place of your real tool calls")UNCERTAIN = ("don't know", "do not know", "couldn't find", "could not find", "not sure", "no information", "unable to find",             "can't find", "cannot find", "?")SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|is done|all set|been (sent|created|saved|submitted|placed)|successfully|went through)\b", re.I)NUM = re.compile(r"\d[\d,]*(?:\.\d+)?")@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    res = run_agent(case["input"], tools)    out = res["output"]    k = case["kind"]    if k in ("error", "timeout"):        assert not SUCCESS.search(out), "agent claimed success after the tool failed"        if k == "timeout":            assert res.get("latency_s", 0) <= 60    elif k == "loop":        counts = {}        for c in tools.calls:            counts[c] = counts.get(c, 0) + 1        assert max(counts.values(), default=0) < 3    elif k == "high_cost":        assert res.get("cost", 0) <= 0.50    elif k == "empty_tool_output":        assert any(u in out.lower() for u in UNCERTAIN), "answer should admit it found nothing or ask a question"    elif k == "tool_output_ignored":        facts = case["expect_facts"]        hit = [f for f in facts if f.lower() in out.lower()]        assert len(hit) >= max(1, (len(facts) + 1) // 2), f"answer ignores the tool result; expected most of {facts}"    elif k == "malformed_tool_output":        seen = json.dumps(case["input"], default=str) + json.dumps([t["output"] for t in case["tool_outputs"]], default=str)        known = {n.replace(",", "") for n in NUM.findall(seen)}        made_up = [n for n in NUM.findall(out) if n.replace(",", "") not in known]        assert not made_up, f"answer states figures not in the input or any tool output: {made_up}"

The real pytest export format. Downloading JSON or pytest files is a Team and Agency feature; Free can view generated cases in the dashboard.

Synthetic sample run

Walkthrough: a tool returns an error page, the agent promises a refund

One run from the sample project, step by step, exactly as Rootline processes it. Every value below is read from the same data as the workspace above.

  1. The run. A customer writes: My blender arrived broken. Can I get a refund and a replacement? The agent (refund-agent) calls get_order, search_policy, refunds_api.check_eligibility across 7 steps.
  2. Tools return unexpected results. Step 04 search_policy returns {"hits": []}. Step 06 refunds_api.check_eligibility returns a 503 HTML error page instead of JSON.
  3. The agent decides wrongly. Its final answer: Good news: you're eligible for a full refund of $349, and a replacement Blender X200 ships tomorrow. No tool output contains that amount or date.
  4. Inspect the trace. In the timeline the failing step is marked; in Raw JSON its lines are highlighted. Open this run in the workspace.
  5. Rootline identifies the failure. Malformed tool output (medium), also detected: Empty tool output. It is grouped into the issue Malformed tool output in refunds_api.check_eligibility (refund-agent), 6 runs in this sample project.
  6. Recommended correction. Treat a non-JSON or 5xx response from refunds_api.check_eligibility as a failure: retry once, then tell the customer you'll confirm by email and open a ticket for a human.
  7. Regression test. Rootline generates rootline_malformed_tool_output_1, which replays the recorded tool outputs as mocks. What it checks: the recorded broken output is replayed, and the answer states no figure that isn't in the user input or tool outputs. Downloading the pytest or JSON export is a Team and Agency feature; Free can view generated cases.

Reproduce it in your own account: sign up free, then in Settings choose Use sample project. The same synthetic runs are analysed by the same detectors. In your account the explanation is generated fresh (AI-written when AI explanations are on, rule-based otherwise), so its wording will differ from the pre-generated text shown here.

Grouped into issues

260 failed runs out of 1411 in this sample project fall into 9 issues; the 5 most severe are listed. Rootline groups by failure type, feature and tool. Which customers an issue affects is a Team and Agency feature.

Sample dataIssues · ranked by severity
IssueSeverityRunsCost
Agent loop in check_availability (booking-agent)critical32$13.21
Agent loop in create_shipment (order-agent)critical5$0.48
Empty tool output in search_kb (support-bot)high62$0.50
Tool output ignored in get_order (order-agent)high50$0.40
Error raised in crm.create_lead (lead-capture)high34$0.27

Sample dataTeam and Agency feature

Cost per customer, and a runaway-loop alert

Total cost across the 1411 sample runs: $39.65. Team and Agency break it down by customer, feature and bot or API key. Free shows overall totals and daily spend only.

Top customers by cost
CustomerRunsCost
harbor-insurance117$4.92
northwind-legal122$4.48
brightpath-realty116$3.99
acme-dental123$3.86
zen-fitness130$3.64
Most expensive single runs
CustomerFeatureRun cost
harbor-insurancereport-writer$1.00
qatar-motors-demoreport-writer$0.94
northwind-legalreport-writer$0.93
pinecrest-hotelsreport-writer$0.93
acme-dentalreport-writer$0.82
critical
Runaway loop: run sample-0184 (booking-agent, user cobalt-logistics) cost $0.24
Alert example · agent loop. The booking-agent loop issue in this sample project covers 32 runs and $13.21. Team and Agency get an email when an agent loops or spend spikes.

What you just saw, and what is different on your data

Sample project

Everything on this page comes from 1411 synthetic runs in Rootline's built-in sample project. It is not real customer data. The analysis is the same code that runs on your traces.

Explanations

Most examples here show Rootline's built-in rule-based explanations. The messy refund example shows an AI-written explanation, generated ahead of time for this sample: this page never calls an AI provider. After you sign up, your failed runs get AI-written explanations from a third-party AI provider. You can turn that off in your workspace settings, and then nothing from your traces is sent.

Your traces

Connect Langfuse with your project keys (no code changes), or point an OpenTelemetry OTLP/HTTP exporter at Rootline. The Free plan includes AI explanations for 100 failed runs per month.

Run this on your own traces

Connect Langfuse or send OpenTelemetry traces and see your own failed runs explained, grouped and turned into tests.

Start debugging free creates a free account, no card. Then connect Langfuse with your project keys or send OpenTelemetry traces.