AI agent debugging tools fall into three layers. Tracing (Langfuse, or any OpenTelemetry backend) records every model call and tool call, so you can see what happened in a run. Evaluations (scores, LLM-as-a-judge, code evaluators, datasets and experiments) measure quality against criteria you define. Failure analysis reads the traces for you and answers which runs failed, which step broke, and whether it is the same problem as yesterday.
Start with tracing: nothing else works without it. Add evaluations for quality you can write down. Add failure analysis once there are more failed runs than anyone can read.
What each layer finds
The layers overlap, but each is built to answer a different question. Knowing which question you are asking saves a lot of scrolling through traces.
| Layer | Answers | Good at | Not built for |
|---|---|---|---|
| Tracing | What happened in this run? | Step-by-step record: inputs, outputs, latency, tokens, cost, errors, sessions and users | Telling you which of 10,000 runs to open |
| Evaluations | How good was this output? | Scoring quality you can define: correctness, tone, format, policy checks, regression gates on datasets | Failures you did not think to write a check for |
| Failure analysis | Which runs broke, where, and why? | Flagging common agent failure patterns, pointing at the failing step, grouping repeats into issues | Judging subjective quality, or anything not visible in the trace |
Dashboards and alerts sit across all three: they tell you that a number moved (error rate, cost, latency), not why.
A support agent handles refund requests with three tools: lookup_order, issue_refund and send_email. A user writes in: "the bot said my refund was processed, but nothing arrived". The run has no exception in the application logs, and the dashboard error rate looks normal.
A debugging workflow that works with any tool
- Make sure tool calls are their own steps in the trace. Each tool call should be a separate observation or span with the tool name, the input arguments, the output, and an error level or status when it fails. If the whole run is one span, no tool can show you where it broke.
- Find the run. Filter by user, session or time window. In Langfuse, the observations table can be filtered by level, name, user and session; with OpenTelemetry, search your backend by trace ID or attributes such as
enduser.id. - Read the failing step, not the final answer. The answer says the refund went through. The step that matters is
issue_refund: its level isERRORand its status message is401 Unauthorized: token expired. - Name the failure type. Here it is an error followed by a false success: the tool failed and the agent still told the user it worked. Other common types are timeouts, loops, empty or malformed tool output, tool output the agent ignored, and unusually expensive runs.
- Check whether it repeats. Group failed runs by failure type, feature and tool. Ten runs with "
issue_refund→ 401" are one bug, not ten. - Fix it and lock it in. Refresh the token before calling the payment API, never let the agent confirm an action whose tool call failed, and add a regression test that replays the recorded 401 and checks the answer does not claim success.
What the trace shows
Simplified observations from the synthetic run above. The highlighted line is the step that failed.
trace refund-request latency 4.1s cost $0.012 GENERATION llm.chat plan: lookup order, then refund TOOL lookup_order {"order_id": "A-1042"} -> {"status": "delivered", "total": 59.0}TOOL issue_refund {"order_id": "A-1042"} level=ERROR "401 Unauthorized: token expired" GENERATION llm.chat "Done! Your refund of $59.00 went through."
The last line is the part a user sees. Without the issue_refund step recorded separately, with its level and status message, the run looks like a success.
Common mistakes
- Debugging from the final answer only. The answer is a symptom. The cause is almost always in a tool step.
- Treating "no exception" as "no failure". Many agent failures never raise: empty search results, an ignored tool result, a loop that ends at the step limit.
- Writing evaluations before you know the failure modes. Read a sample of failed runs first, then write checks for what you actually saw.
- Fixing one run at a time. Group by failure type and tool first; the biggest group is usually the best fix.
- Not keeping the failed run. Without the recorded inputs and tool outputs you cannot write a test that reproduces it.
When Rootline helps
Rootline is the failure-analysis layer. It reads your Langfuse or OpenTelemetry traces, flags failed and suspicious runs (errors, timeouts, agent loops, empty or malformed tool output, tool output the agent ignored, and runs over $0.50), points at the failing step, and writes a plain-English explanation with a suggested fix. When a tool call failed and the final answer still claims success (for example "booked", "went through" or "successfully"), the built-in explanation calls it out as a false success. Repeats are grouped into one issue by failure type, feature and tool.
Rootline does not replace tracing or evaluations: it needs traces from Langfuse or OpenTelemetry, and it does not score subjective quality such as tone or helpfulness. Its checks are rules over what the trace contains, so a wrong answer built from well-formed tool results, with no error, loop or empty output, will not be flagged. Use evaluations for that.
Official documentation
- Langfuse observability overviewlangfuse.com
- Langfuse evaluation overviewlangfuse.com
- Langfuse log levels (ERROR, WARNING) and status messageslangfuse.com
- OpenTelemetry traces conceptsopentelemetry.io
- OpenTelemetry semantic conventions for generative AIopentelemetry.io