For teams using Langfuse & OpenTelemetry

Know why your AI agent failed.

Stop investigating the same failure across dozens of traces. Rootline groups repeated failures into issues, identifies the likely cause and suggests a fix to test — using the traces you already collect.

  • Langfuse + OpenTelemetry
  • No agent rewrite for Langfuse
  • Free to start

Example diagnosisThe shipping API kept returning a 503, and the agent retried the identical call until it gave up.

Synthetic sample runorder-agentsample-140615.4s9 steps$0.072

Trace

user
Please ship order C-3318 by express today.
agent
Sorry, I couldn't create the shipment for order C-3318 right now. Please try again later.
  1. 01LLMllm3.14s
  2. 02Toolcreate_shipment503 carrier_unavailablecall 1/41.56s
  3. 03LLMllm1.20s
  4. 04Toolcreate_shipment503 carrier_unavailablecall 2/40.19s
  5. 05LLMllm1.18s
  6. 06Toolcreate_shipment503 carrier_unavailablecall 3/40.94s
  7. 07LLMllm2.48s
  8. 08Toolcreate_shipment503 carrier_unavailablecall 4/41.24s
  9. 09LLMllm2.06s

4× create_shipmentidentical input4 of 4 returned 503

Step 02 · create_shipment · output{"error": {"status": 503, "code": "carrier_unavailable", "message": "Service Unavailable: upstream carrier API did not respond"}}

Rootline diagnosisRule-based

Failure

Agent loopcritical

Grouped into Agent loop in create_shipment (order-agent) · 5 runs

Evidencefrom the trace

create_shipment called 4x with identical input; every call returned 503 carrier_unavailable.

Steps 02, 04, 06, 08. The tool span isn't marked as an error, so the 503 only shows in its output.

Likely causeRootline's explanation

The agent got stuck in a loop: create_shipment called 4x with identical input. It kept retrying the same step instead of answering, burning tokens.

Suggested fixa change to test

Cap repeated identical tool calls (e.g. max 2), and tell the model in the prompt to stop and answer or escalate when a tool repeats the same result.

Sample project1,411 runs260 failed · 18.4%81.6% healthy9 issues$39.65 LLM costOpen this run in the demo →

Rendered from Rootline's built-in synthetic sample project with the same analysis code that runs on your traces — not a real customer workspace.

Why Rootline

Your traces show what happened. Rootline helps you decide what to fix.

Keep the tracing you already have. Rootline reads the same traces and adds a failure-first workflow: which problems keep happening, why, and what to test.

Your tracing stack

What happened, step by step

  • Langfuse already covers tracing, debugging and evaluations, with observability dashboards for cost, latency and quality. You can open any trace, follow a session and score outputs there.
  • OpenTelemetry is the open standard for instrumenting your agent and exporting its traces.

Rootline doesn't replace either. It reads Langfuse through its API, or receives the OTLP traces you export to it.

Rootline adds

What to fix, and how to check the fix

  1. Group recurring failures into issuesRuns with the same failure type, feature and tool become one issue with a run count, cost and severity.
  2. Surface the failing step and its evidenceErrors, timeouts, loops, empty or malformed tool output and ignored results, with the input, output or error from the trace.
  3. Explain the likely cause and suggest a fixA plain-English cause and a change to test, labelled rule-based or AI-written.
  4. Export regression testsA test case generated from the failed run, exported as JSON or pytest on Team and Agency.

Evidence

What the trace recorded: the span, its input and output, the error.

Likely cause

Rootline's explanation of that evidence. A diagnosis, not a certainty.

Suggested fix

A change to try, then check with the generated regression test.

Failure grouping

260 failed runs. 9 issues to review.

Failures with the same failure type, feature and tool become one issue, so you investigate a pattern once instead of reading every trace. Open an issue to see its likely cause.

Synthetic sample project1,411 runs · 260 failed · 9 issues

Failed runs → issues

260failed runs
9issues, ranked by severity
Failed runs by severity

Counted from the synthetic sample project: 1,411 runs, 18.4% failed. The issue counts add up to 260.

  1. sample-1367sample-1316+30 more

    Likely causeThe agent got stuck in a loop: check_availability called 8x with identical input. It kept retrying the same step instead of answering, burning tokens.

    Cost$13.21 across 32 runs

    Inspect an example run →
  2. sample-1407sample-1406+3 more

    Likely causeThe agent got stuck in a loop: create_shipment called 4x with identical input. It kept retrying the same step instead of answering, burning tokens.

    Cost$0.48 across 5 runs

    Inspect an example run →
  3. sample-1309sample-0007+60 more

    Likely causesearch_kb returned an empty result, and the agent answered anyway instead of saying it didn't know, so the answer is likely made up.

    Cost$0.50 across 62 runs

    Inspect an example run →
  4. sample-0043sample-0232+48 more

    Likely causeget_order returned the right information, but the agent's final answer doesn't use it (final answer uses none of get_order's returned facts). The model ignored its own tool result.

    Cost$0.40 across 50 runs

    Open the sample demo →
  5. sample-0621+33 more

    Likely causeThe run failed because crm.create_lead rejected the agent's credentials (HTTP 401 Unauthorized: token expired). Worse, the agent then told the user "Thanks! Your viewing is booked." even though the call failed: a false success, so the user thinks it worked and nobody follows up.

    Cost$0.27 across 34 runs

    Inspect an example run →
  6. sample-0474+26 more

    Likely causeThe run was too slow: Request timed out after 90s. The user most likely waited and got no useful answer.

    Cost$0.22 across 27 runs

    Open the sample demo →
  7. Likely causepricing_api returned output the agent couldn't parse (pricing_api returned unparseable output), so the answer was built on broken data.

    Cost$0.21 across 26 runs

    Open the sample demo →
  8. Likely causerefunds_api.check_eligibility returned output the agent couldn't parse (refunds_api.check_eligibility returned unparseable output), so the answer was built on broken data.

    Cost$0.14 across 6 runs

    Inspect an example run →
  9. Likely causeThis run was unusually expensive: run cost $0.74 (> $0.50).

    Cost$13.71 across 18 runs

    Inspect an example run →
Customers affected per issue: Team and AgencyOpen the full demo →

Product

Catch the quiet failures, and see what they cost

Every step of each failed or suspicious run is checked, so failures that don't raise an error still surface, with what each run cost.

01 · Find the failure

The failing step, the likely cause, and the evidence

Errors, timeouts, loops, empty or malformed tool output and ignored tool results are detected step by step. Each failure gets an explanation that points at the span where it went wrong.

It also catches the quiet ones: here a CRM call returned 401, and the agent told the user the booking worked.

Samplelead-capture·Error raised
  1. 01LLMllm 1.83s
  2. 02Toolcrm.create_leadfailing step 0.16s
  3. 03LLMllm 3.08s
status
HTTP 401 Unauthorized: token expired
agent
Thanks! Your viewing is booked.

The run failed because crm.create_lead rejected the agent's credentials (HTTP 401 Unauthorized: token expired). Worse, the agent then told the user "Thanks! Your viewing is booked." even though the call failed: a false success, so the user thinks it worked and nobody follows up.

02 · Understand the cost

What failures cost, and who they cost

Every plan shows overall LLM cost totals and daily spend. Team and Agency break cost down by customer, feature and bot or API key, and send an email when an agent loops or spend spikes.

The booking-agent loop issue in this sample covers 32 runs and $13.21.

SampleCosts · 1,411 runs
$39.65Total LLM cost · all plans
1,411Runs
Top customers by LLM cost (USD)Team and Agency
criticalRunaway loop: run sample-0184 (booking-agent, user cobalt-logistics) cost $0.24

Regression tests

Turn a failed run into a test you run

Rootline generates a regression-test starting point from a failed run. It replays the run's recorded tool outputs as mocks, so the test is deterministic and needs no live APIs.

  1. Failed run

    sample-1406 · order-agent
    Agent loop in create_shipment (order-agent)
  2. Suggested fix

    Cap repeated identical tool calls (e.g. max 2), and tell the model in the prompt to stop and answer or escalate when a tool repeats the same result.
  3. Export test

    JSON or pytest, Team and Agency. Free can view generated cases in the dashboard.
  4. Run in your test suite

    Connect run_agent to your agent, then run it with pytest locally or in CI.
Samplerootline_loop_1

Excerpt of test_rootline_regressions.py generated for this run: the check that fails if the agent loops again.

@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    res = run_agent(case["input"], tools)    out = res["output"]    k = case["kind"]    ...    elif k == "loop":        counts = {}        for c in tools.calls:            counts[c] = counts.get(c, 0) + 1        assert max(counts.values(), default=0) < 3
"""Regression tests generated by Rootline from real failed runs.Each case replays the tool outputs recorded in the failing run, so the test is deterministic and needs no live APIs.Wire `run_agent(user_input, tools)` to your agent: route every tool call through `tools(name, input)` andreturn dict(output=str, latency_s=float, cost=float)."""import json, re, pytestCASES = json.loads(r"""[  {    "id": "rootline_loop_1",    "issue": "Agent loop in create_shipment (order-agent)",    "source_run": "sample-1406",    "feature": "order-agent",    "input": "Please ship order C-3318 by express today.",    "bad_output": "Sorry, I couldn't create the shipment for order C-3318 right now. Please try again later.",    "kind": "loop",    "tool_outputs": [      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      }    ],    "assertions": [      "no tool is called 3 or more times with identical input (counted on the replayed tools)"    ]  }]""")class ToolError(Exception):    passclass Replay:    """Mock tools: returns the outputs recorded in the failing run, in order per tool name (repeating the last one)."""    def __init__(self, recorded):        self.recorded, self.calls, self._pos = recorded, [], {}    def __call__(self, name, tool_input=""):        self.calls.append((name, json.dumps(tool_input, sort_keys=True, default=str)))        mine = [t for t in self.recorded if t["name"] == name]        if not mine:            raise ToolError(f"tool {name!r} was not called in the recorded run")        i = self._pos.get(name, 0)        self._pos[name] = i + 1        t = mine[min(i, len(mine) - 1)]        if t.get("error") and "timed out" in t["error"].lower():            raise TimeoutError(t["error"])        if t.get("level") == "ERROR":            raise ToolError(t.get("error") or "tool error")        return t["output"]def run_agent(user_input, tools):    raise NotImplementedError("Connect this to your agent entrypoint and pass `tools` in place of your real tool calls")UNCERTAIN = ("don't know", "do not know", "couldn't find", "could not find", "not sure", "no information", "unable to find",             "can't find", "cannot find", "?")SUCCESS = re.compile(r"\b(booked|confirmed|scheduled|is done|all set|been (sent|created|saved|submitted|placed)|successfully|went through)\b", re.I)NUM = re.compile(r"\d[\d,]*(?:\.\d+)?")@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])def test_regression(case):    tools = Replay(case["tool_outputs"])    res = run_agent(case["input"], tools)    out = res["output"]    k = case["kind"]    if k in ("error", "timeout"):        assert not SUCCESS.search(out), "agent claimed success after the tool failed"        if k == "timeout":            assert res.get("latency_s", 0) <= 60    elif k == "loop":        counts = {}        for c in tools.calls:            counts[c] = counts.get(c, 0) + 1        assert max(counts.values(), default=0) < 3    elif k == "high_cost":        assert res.get("cost", 0) <= 0.50    elif k == "empty_tool_output":        assert any(u in out.lower() for u in UNCERTAIN), "answer should admit it found nothing or ask a question"    elif k == "tool_output_ignored":        facts = case["expect_facts"]        hit = [f for f in facts if f.lower() in out.lower()]        assert len(hit) >= max(1, (len(facts) + 1) // 2), f"answer ignores the tool result; expected most of {facts}"    elif k == "malformed_tool_output":        seen = json.dumps(case["input"], default=str) + json.dumps([t["output"] for t in case["tool_outputs"]], default=str)        known = {n.replace(",", "") for n in NUM.findall(seen)}        made_up = [n for n in NUM.findall(out) if n.replace(",", "") not in known]        assert not made_up, f"answer states figures not in the input or any tool output: {made_up}"

The same case as it appears in rootline_regressions.json (formatted): the input, the bad output and the 4 recorded tool outputs it replays.

[  {    "id": "rootline_loop_1",    "issue": "Agent loop in create_shipment (order-agent)",    "source_run": "sample-1406",    "feature": "order-agent",    "input": "Please ship order C-3318 by express today.",    "bad_output": "Sorry, I couldn't create the shipment for order C-3318 right now. Please try again later.",    "kind": "loop",    "tool_outputs": [      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      },      {        "name": "create_shipment",        "input": {          "order_id": "C-3318",          "service": "express",          "parcels": 2,          "ship_to": "addr_7741"        },        "output": {          "error": {            "status": 503,            "code": "carrier_unavailable",            "message": "Service Unavailable: upstream carrier API did not respond"          }        },        "level": "DEFAULT",        "error": null      }    ],    "assertions": [      "no tool is called 3 or more times with identical input (counted on the replayed tools)"    ]  }]
Checks
no tool is called 3 or more times with identical input (counted on the replayed tools).
You configure
Implement run_agent(user_input, tools) so your agent's tool calls go through the replayed tools.
Not covered
Live APIs, other inputs and answer quality beyond the check above.
Who runs it
You do, in your own test suite or CI. Rootline generates the test; it doesn't run it, so no pass or fail result is shown here.

Plans. Regression test export (JSON and pytest download) is on Team and Agency. Free can view the generated cases in the dashboard.

Setup

How it works with Langfuse and OpenTelemetry

No data migration. Keep your tracing as it is: Langfuse needs no code changes; OpenTelemetry needs a few lines of exporter config.

  1. Create a free workspace

    No credit card. Load a labelled sample project with one click if you want to look around before connecting your own traces.

  2. Connect your traces

    Langfuse: paste host + API keys in Settings. OpenTelemetry: point your OTLP/HTTP exporter at Rootline with an ingest key. Use either, or both — Langfuse and OpenTelemetry together in one list of runs.

  3. Inspect failures

    Failed and suspicious runs get a plain-English reason and a suggested fix, with the failing step highlighted. AI explanations appear within about 1–3 minutes of a run syncing.

Langfuseno code changes

In Langfuse, create an API key pair (Settings → API Keys). In Rootline, paste the host and both keys. Rootline then reads your traces and observations; the first import covers the last 30 days. Self-hosted Langfuse works too.

host        https://cloud.langfuse.compublic key  pk-lf-…secret key  sk-lf-…   # encrypted at rest
OpenTelemetryexporter config

Create an ingest key in Settings, then send OTLP/HTTP traces (protobuf or JSON) to Rootline. GenAI and OpenInference semantic-convention spans work as they are.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://rootline.site/v1/tracesOTEL_EXPORTER_OTLP_PROTOCOL=http/protobufOTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<ingest-key>"

Your data, question by question

Rootline sits next to production tracing, so here is exactly what it does with your traces, checked against how the product works today. Full details are in the privacy policy.

Can Rootline change my Langfuse data?
Rootline only makes read requests to Langfuse. Langfuse project API keys themselves allow more than reading, so treat them as sensitive: the secret key is encrypted before storage and you can revoke the pair in Langfuse at any time.
What is sent to an AI provider?
Only for failed or suspicious runs: a redacted excerpt of about 2,000 characters (the failing step's input, output and error, the user input and final answer, plus step names, failure type, cost and latency). Full traces are not sent. Redaction is pattern-based and can miss unusual formats or names.
When does that happen, and can I turn it off?
Automatically, shortly after new failed runs arrive from a Langfuse sync or an OpenTelemetry export, never just because you open a page. Turn AI explanations off in Settings and nothing from your traces is sent to any AI provider.
Are explanations always written by AI?
No. Explanations marked "AI" were written by AI. Otherwise, or when AI is off, unavailable or past your monthly allowance, you get Rootline's built-in rule-based explanation.
How long is data kept?
Trace data, explanations and alert records are deleted automatically once they are more than 90 days old.
Where is it stored?
In the United States. There is no EU data region yet. Each workspace's data is isolated, and OpenTelemetry traces are accepted only with your workspace's ingest key.
Can I delete my data?
Disconnecting Langfuse deletes its stored keys and the runs imported from it. Deleting your account in Settings deletes your workspace's runs, spans, explanations, keys and settings.
Do healthy runs count?
No. Only failed or suspicious runs get an explanation, and only those count toward your AI allowance.

Pricing

Simple, flat pricing

Priced by failed runs analyzed with AI explanations per month. Healthy runs are never counted.

Free

Try Rootline on your own traces

$0/month

No credit card

100 AI explanations/mo

  • Issues, explanations, run timeline
  • Overall cost totals and daily spend
  • Langfuse and OpenTelemetry together
  • Rule-based explanations after the allowance
  • Regression test export
  • Cost per customer, customer view, alerts
Start free

Team

For production teams: test fixes, see who is affected, get alerted

$49/month

Billed monthly · cancel anytime

2,000 AI explanations/mo

  • Everything in Free
  • Regression test export (JSON and pytest)
  • Cost per customer and feature
  • Customer view
  • Email alerts for agent loops and cost spikes
Get Team

Agency

The higher-volume option

$199/month

Billed monthly · cancel anytime

10,000 AI explanations/mo

  • Everything in Team
  • 5× Team's AI allowance, for higher volumes
  • Client workspaces and white-label reports (not yet)
Get Agency

What countsA failed run counts once, when it first gets an AI explanation. Healthy runs are never counted, and a repeat of an already-explained failure reuses the stored explanation.

When the allowance runs outPast your AI allowance, failed runs still appear with the built-in rule-based explanation. AI explanations resume at the start of the next month (UTC) or after an upgrade.

BillingPaid plans are monthly and billed securely by our payment provider, acting as merchant of record. Cancel anytime from Billing.

FAQ

Frequently asked questions

Something else? Email support@rootline.site.

No. Rootline works on top of the Langfuse or OpenTelemetry traces you already have. Keep your tracing stack; Rootline explains failures and cost on top.

Langfuse: no. Rootline reads the traces you already send to Langfuse, including self-hosted Langfuse. OpenTelemetry: some setup. If your agent already emits OpenTelemetry spans, point your OTLP/HTTP exporter at Rootline with your ingest key (a few lines of exporter config); GenAI semantic-convention spans work as they are. If it isn't instrumented yet, add OpenTelemetry instrumentation first.

Langfuse and OpenTelemetry (OTLP/HTTP traces and logs; GenAI and OpenInference conventions). You can connect both in the same workspace. Other tracing tools are planned but not available yet.

Errors, timeouts, loops, empty or malformed tool output, tool results the agent ignored, and unusually expensive runs.

Rootline finds the failing step and writes a plain-English explanation of why the run failed, the likely cause, and a suggested fix. Explanations are written by a third-party AI provider and appear within about 1–3 minutes of a run syncing. If AI is unavailable, Rootline falls back to its built-in rule-based explanation, so every failure still gets one. Explanations marked "AI" were written by AI.

To write AI explanations, Rootline sends a third-party AI provider a short, truncated excerpt of each failed run: the failing step's input, output and error, plus the run's user input and final answer, about 2,000 characters in total (along with step and tool names, the failure type, cost and latency). Before sending, Rootline automatically redacts text that matches common patterns for API keys, tokens and email addresses; pattern-based redaction can miss unusual formats, and other personal data in your traces, such as names, may not be removed. The provider processes this data only to generate the explanation; under its API terms, data sent through the API is not used to train its models by default. Full traces are not sent. You can turn AI explanations off for your workspace in Settings. See our privacy policy.

Free: issues, explanations, run timeline, overall cost totals and daily spend, Langfuse and OpenTelemetry together, and 100 AI-explained failed runs a month. Team adds regression test export, cost per customer and feature, the customer view on issues and runs, and email alerts, with 2,000 AI-explained failed runs a month. Agency ($199/mo) has the same features as Team with 10,000 AI-explained failed runs a month.

Nothing breaks. New failed runs still sync, are grouped into issues and get the built-in rule-based explanation. AI explanations start again at the beginning of the next calendar month (UTC), or after you move to a plan with a higher allowance. Healthy runs never count, and a failure that was already explained reuses its stored explanation.

Each workspace's data is isolated. Langfuse project API keys give access to your whole project, but Rootline only uses them to read traces and observations and never writes to or changes your Langfuse data. Your secret key is encrypted before storage, and you can revoke the keys in Langfuse at any time. OpenTelemetry traces are accepted only with your workspace's ingest key and stored in your workspace. Trace data is deleted automatically after 90 days.

See why your last failed run broke.

Start on the sample project, then connect Langfuse or OpenTelemetry when you are ready.