Guide

Silent AI agent failures: when a run fails without an error

The hardest failures to find are the ones that look like success: the run finishes, no step is marked as an error, and the answer is wrong. They need different checks from exceptions.

RootlinePublished

Short answer

A silent AI agent failure is a run that finishes, has no step marked as an error, and still gives the user a wrong or empty result. The common kinds are: a tool returned nothing and the agent answered anyway; a tool returned broken output (an HTML error page, a stack trace, cut-off JSON) as a normal string; the agent ignored what the tool returned; and the agent told the user an action succeeded when it didn't. You find them by checking tool outputs and the final answer against them, not by filtering on error level.

Four kinds of silent failure

KindIn the traceWhat the user gets
Empty tool outputOutput is null, "", [], {} or {"hits": []}; level is normalAn invented answer, or a vague one
Malformed tool outputA string that starts like JSON but doesn't parse, or contains Traceback (most recent call last) or <!DOCTYPE html>Numbers or facts the model guessed from noise
Tool output ignoredThe tool returned specific facts (IDs, amounts, dates); the final answer contains none of themA generic answer that doesn't use the data it fetched
False successAn action tool failed (or returned nothing) and the answer says "booked", "sent", "done"A confirmation for something that didn't happen
Synthetic example · not customer data

An internal HR assistant answers policy questions using a search_policies tool. After a search index migration, the tool returns {"hits": []} for most queries. Nothing errors. The assistant keeps answering from general knowledge, and employees get confident answers that don't match the company's actual policy.

How to find them

  1. Record tool outputs in full. If your traces store only "tool called", none of these checks are possible. Make sure inputs and outputs of tool steps are captured (with sensitive fields redacted as needed).
  2. Scan tool outputs for empty and broken values. Count tool steps whose output is empty, or a string that fails to parse as JSON when it should be JSON, per tool per day. A jump after a deploy or migration is your signal.
  3. Compare the answer with the last tool result. Pull the specific facts out of the tool output (IDs, numbers, longer words) and check whether any appear in the final answer.
  4. Look for success claims after failed actions. If an action tool failed or returned nothing and the final answer contains "booked", "confirmed", "sent" or "successfully", the user was told something false.
  5. Read a sample by hand. Automatic checks find candidates; ten minutes reading flagged runs tells you which checks are too noisy.
  6. Fix the tool contract. Make empty results explicit ("no results; tell the user"), return errors as errors instead of strings, and validate tool output against a schema before it reaches the model.
trace policy-question                 latency 2.3s   cost $0.006  GENERATION llm.chatTOOL       search_policies {"q": "parental leave weeks"}  -> {"hits": []}  GENERATION llm.chat   "Employees get 16 weeks of paid parental leave."

Checks you can write

import json, redef is_empty(v):    if isinstance(v, dict) and v and all(is_empty(x) for x in v.values()):        return True                                   # {"hits": []}    return v in (None, "", [], {}) or (isinstance(v, str) and v.strip() in ("null", "[]", "{}"))def is_malformed(v):    if not isinstance(v, str):        return False    s = v.strip()    if s.startswith(("{", "[")):        try:            json.loads(s); return False        except ValueError:            return True    return bool(re.search(r"Traceback \(most recent|<!DOCTYPE html|<html", s, re.I))def facts(v):    txt = v if isinstance(v, str) else json.dumps(v)    return {t.lower() for t in re.findall(r"[A-Za-z0-9][A-Za-z0-9.$-]{3,}", txt)            if any(c.isdigit() for c in t) or len(t) >= 6}def ignored(last_tool_output, answer):    f = facts(last_tool_output)    return len(f) >= 3 and not f & facts(answer)

These are deliberately simple and lexical. They catch the obvious cases cheaply; for "is this answer actually right?", use an evaluation (a code evaluator with your own rules, or an LLM-as-a-judge score).

Common mistakes

  • Only alerting on errors. Error rate can be flat while half the answers are made up.
  • Tools that swallow exceptions and return "Error: something went wrong" as a normal result.
  • Not telling the model what empty means. A bare [] invites guessing.
  • Letting the agent confirm actions without checking the action tool succeeded.
  • Judging by the final answer only. A fluent answer looks fine until you compare it with what the tool returned.

When Rootline helps

Where it fits

Rootline checks every tool step for Empty tool output (null, empty string, list or object, the strings "null", "[]" and "{}", or an object whose values are all empty) and Malformed tool output (text that starts like JSON but doesn't parse, or contains a Python traceback or an HTML page). It flags Tool output ignored when the last tool's output contains at least three specific facts and the final answer uses none of them. When a tool call errored and the answer still claims success, the built-in explanation calls out the false success. Each flagged run gets the step highlighted and a plain-English explanation with a fix.

Limitations

The ignored-output check is lexical, not semantic: an answer that paraphrases every fact can be flagged, and one that repeats a single irrelevant token passes. Only the last tool step is compared with the answer, and the check is skipped when the run is already a loop. A tool that returns well-formed but wrong data is not detected. These checks complement evaluations; they don't replace them.

Official documentation

Find out why your last failed run broke

Rootline reads your Langfuse or OpenTelemetry traces and explains each failed run: the failing step, the likely cause and a suggested fix. Free to start, no credit card.

Start debugging free creates a free account, no card. Then connect Langfuse or OpenTelemetry.