A silent AI agent failure is a run that finishes, has no step marked as an error, and still gives the user a wrong or empty result. The common kinds are: a tool returned nothing and the agent answered anyway; a tool returned broken output (an HTML error page, a stack trace, cut-off JSON) as a normal string; the agent ignored what the tool returned; and the agent told the user an action succeeded when it didn't. You find them by checking tool outputs and the final answer against them, not by filtering on error level.
Four kinds of silent failure
| Kind | In the trace | What the user gets |
|---|---|---|
| Empty tool output | Output is null, "", [], {} or {"hits": []}; level is normal | An invented answer, or a vague one |
| Malformed tool output | A string that starts like JSON but doesn't parse, or contains Traceback (most recent call last) or <!DOCTYPE html> | Numbers or facts the model guessed from noise |
| Tool output ignored | The tool returned specific facts (IDs, amounts, dates); the final answer contains none of them | A generic answer that doesn't use the data it fetched |
| False success | An action tool failed (or returned nothing) and the answer says "booked", "sent", "done" | A confirmation for something that didn't happen |
An internal HR assistant answers policy questions using a search_policies tool. After a search index migration, the tool returns {"hits": []} for most queries. Nothing errors. The assistant keeps answering from general knowledge, and employees get confident answers that don't match the company's actual policy.
How to find them
- Record tool outputs in full. If your traces store only "tool called", none of these checks are possible. Make sure inputs and outputs of tool steps are captured (with sensitive fields redacted as needed).
- Scan tool outputs for empty and broken values. Count tool steps whose output is empty, or a string that fails to parse as JSON when it should be JSON, per tool per day. A jump after a deploy or migration is your signal.
- Compare the answer with the last tool result. Pull the specific facts out of the tool output (IDs, numbers, longer words) and check whether any appear in the final answer.
- Look for success claims after failed actions. If an action tool failed or returned nothing and the final answer contains "booked", "confirmed", "sent" or "successfully", the user was told something false.
- Read a sample by hand. Automatic checks find candidates; ten minutes reading flagged runs tells you which checks are too noisy.
- Fix the tool contract. Make empty results explicit ("no results; tell the user"), return errors as errors instead of strings, and validate tool output against a schema before it reaches the model.
trace policy-question latency 2.3s cost $0.006 GENERATION llm.chatTOOL search_policies {"q": "parental leave weeks"} -> {"hits": []} GENERATION llm.chat "Employees get 16 weeks of paid parental leave."
Checks you can write
import json, redef is_empty(v): if isinstance(v, dict) and v and all(is_empty(x) for x in v.values()): return True # {"hits": []} return v in (None, "", [], {}) or (isinstance(v, str) and v.strip() in ("null", "[]", "{}"))def is_malformed(v): if not isinstance(v, str): return False s = v.strip() if s.startswith(("{", "[")): try: json.loads(s); return False except ValueError: return True return bool(re.search(r"Traceback \(most recent|<!DOCTYPE html|<html", s, re.I))def facts(v): txt = v if isinstance(v, str) else json.dumps(v) return {t.lower() for t in re.findall(r"[A-Za-z0-9][A-Za-z0-9.$-]{3,}", txt) if any(c.isdigit() for c in t) or len(t) >= 6}def ignored(last_tool_output, answer): f = facts(last_tool_output) return len(f) >= 3 and not f & facts(answer)
These are deliberately simple and lexical. They catch the obvious cases cheaply; for "is this answer actually right?", use an evaluation (a code evaluator with your own rules, or an LLM-as-a-judge score).
Common mistakes
- Only alerting on errors. Error rate can be flat while half the answers are made up.
- Tools that swallow exceptions and return
"Error: something went wrong"as a normal result. - Not telling the model what empty means. A bare
[]invites guessing. - Letting the agent confirm actions without checking the action tool succeeded.
- Judging by the final answer only. A fluent answer looks fine until you compare it with what the tool returned.
When Rootline helps
Rootline checks every tool step for Empty tool output (null, empty string, list or object, the strings "null", "[]" and "{}", or an object whose values are all empty) and Malformed tool output (text that starts like JSON but doesn't parse, or contains a Python traceback or an HTML page). It flags Tool output ignored when the last tool's output contains at least three specific facts and the final answer uses none of them. When a tool call errored and the answer still claims success, the built-in explanation calls out the false success. Each flagged run gets the step highlighted and a plain-English explanation with a fix.
The ignored-output check is lexical, not semantic: an answer that paraphrases every fact can be flagged, and one that repeats a single irrelevant token passes. Only the last tool step is compared with the answer, and the check is skipped when the run is already a loop. A tool that returns well-formed but wrong data is not detected. These checks complement evaluations; they don't replace them.
Official documentation
- Langfuse code evaluators (JSON parseability, schema and tool-call checks)langfuse.com
- Langfuse LLM-as-a-judgelangfuse.com
- What does a good Langfuse trace look like?langfuse.com
- OpenTelemetry GenAI spans: tool call arguments and resultsopentelemetry.io
- JSON Schema referencejson-schema.org