Your Agent Is Hallucinating and Your Dashboard Says Everything Is Fine
Build behavioral observability for LLM agents: structured tracing, LLM-judge evaluation, and closed-loop feedback to catch hallucination and silent failures.
You deployed an LLM agent last month. Latency looks good. Error rate is near zero. Uptime is excellent. Every SLA is green.
And yet, somewhere in production, your agent is confidently citing a policy that does not exist, calling an API with malformed parameters it then silently ignores, or answering a user's medical question with invented dosage information. The HTTP response is 200 OK. Your monitoring never flinched.
This is the silent failure problem, and it is the defining observability challenge of the agentic AI era.
Infrastructure metrics were designed for deterministic systems: a server either responds or it does not, a query either executes or it throws an error. LLM agents are probabilistic reasoning engines. They can fail in ways that produce no exception, no timeout, no anomalous status code, and no visible trace in your existing dashboards. Building reliable agents in production requires a fundamentally different observability model: one that measures behavior rather than infrastructure health alone.
The gap is already costing teams real money. Industry surveys report that 73 percent of enterprises need agent monitoring, yet more than 60 percent lack adequate tooling to do it. The organizations closing that gap are the ones shipping agents that actually work reliably at scale. Here is how to build the same systems they are building.
Why Traditional Monitoring Misses the Point
Standard observability stacks give you logs, metrics, and traces built around a simple mental model: did the system do the thing it was asked to do, and did it do it quickly? For a web server, a database, or a message queue, this model is sufficient.
For an LLM agent, it is dangerously incomplete.
Consider a few failure modes that traditional monitoring cannot catch:
Hallucination without error. An agent researching a customer's compliance question invents a regulation that was repealed two years ago. It returns a well-formatted, confident answer. The response time was 1.2 seconds. Nothing in your logging pipeline sees a problem.
Silent tool misuse. An agent selects the right tool but passes a subtly wrong parameter. The tool executes successfully, returns a result, and the agent incorporates that result into its response without flagging the discrepancy. From the outside, the transaction looks healthy.
Prompt injection through trusted channels. A user pastes a document into a chat agent. The document contains an embedded instruction that redirects the agent's tool calls toward an attacker-controlled endpoint. The agent's outputs look reasonable. The exfiltration happens silently. Adversarial research has demonstrated that the vast majority of implicit prompt injection attacks succeed without triggering output-based safety evaluations.
Each of these is a behavioral failure. None of them looks like a failure to a system that only watches infrastructure. Catching them requires a fundamentally different layer of observability.
The Three-Layer Behavioral Observability Stack
Effective agent observability is not a single tool or a single metric. It is a stack of three cooperating capabilities: instrumentation that captures everything the agent did, evaluation that scores whether what it did was correct, and a feedback loop that turns production failures into future safeguards.
Layer 1: Tracing and Instrumentation
Tracing for agents goes far beyond logging function calls. A useful agent trace records, for every step in a multi-turn reasoning chain: the full prompt sent to the model, the model's response, every tool call made and its result, the retrieval queries issued and the chunks returned, the token counts consumed, and the time elapsed at each stage.
The goal is an execution graph: a directed record of every decision the agent made and every piece of context it had when it made that decision. Without this, root cause analysis is guesswork.
import time
from dataclasses import dataclass, field
@dataclass
class StepTrace:
step_name: str
prompt_tokens: int
completion_tokens: int
tool_calls: list[dict]
retrieval_queries: list[str]
retrieved_chunks: list[str]
model_output: str
duration_ms: float
metadata: dict = field(default_factory=dict)
class AgentTracer:
def __init__(self, run_id: str):
self.run_id = run_id
self.steps: list[StepTrace] = []
def record_step(
self,
step_name: str,
prompt_tokens: int,
completion_tokens: int,
tool_calls: list[dict],
retrieval_queries: list[str],
retrieved_chunks: list[str],
model_output: str,
start_time: float,
metadata: dict | None = None,
) -> StepTrace:
trace = StepTrace(
step_name=step_name,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
tool_calls=tool_calls,
retrieval_queries=retrieval_queries,
retrieved_chunks=retrieved_chunks,
model_output=model_output,
duration_ms=(time.time() - start_time) * 1000,
metadata=metadata or {},
)
self.steps.append(trace)
return trace
def to_dict(self) -> dict:
return {
"run_id": self.run_id,
"total_steps": len(self.steps),
"total_tokens": sum(
s.prompt_tokens + s.completion_tokens for s in self.steps
),
"steps": [vars(s) for s in self.steps],
}
With structured traces like this, you can query across thousands of production runs: find every case where tool_calls was empty but a tool-dependent answer was given, or every case where retrieved_chunks was empty but the response cited specific facts. Those patterns are your silent failures.
Layer 2: Evaluation and Quality Scoring
Traces tell you what happened. Evaluation tells you whether what happened was correct. The key insight is that quality scoring does not have to be manual. You can use an LLM as a judge, applied continuously against production traces, to score agent behavior at each step.
A useful LLM judge evaluates specific behavioral criteria rather than overall impressiveness. For a research agent, those criteria might be: Did the retrieval query actually match the user's intent? Does the reasoning chain logically follow from the retrieved context? Does the final answer avoid making factual claims not supported by the context?
import json
import anthropic
client = anthropic.Anthropic()
JUDGE_PROMPT = """You are evaluating a step in an AI agent's reasoning chain.
Agent task: {task}
Retrieved context: {context}
Agent's reasoning at this step: {reasoning}
Tool calls made: {tool_calls}
Score each criterion from 0 to 1 and explain briefly.
Criteria:
1. retrieval_relevance: Does the retrieved context actually address the task?
2. reasoning_groundedness: Is the reasoning supported by the retrieved context, or does it introduce unsupported claims?
3. tool_selection_correctness: Were the right tools called for this step?
4. hallucination_risk: Does the reasoning contain specific claims not found in the context?
Respond with a JSON object:
{{
"retrieval_relevance": <0-1>,
"reasoning_groundedness": <0-1>,
"tool_selection_correctness": <0-1>,
"hallucination_risk": <0-1>,
"flags": ["list any specific concerns"],
"overall": <0-1>
}}"""
def score_agent_step(
task: str,
context: str,
reasoning: str,
tool_calls: list[dict],
) -> dict:
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=512,
messages=[
{
"role": "user",
"content": JUDGE_PROMPT.format(
task=task,
context=context,
reasoning=reasoning,
tool_calls=json.dumps(tool_calls, indent=2),
),
}
],
)
return json.loads(response.content[0].text)
Run this judge over a sample of production traces every hour. Any step where hallucination_risk exceeds 0.6, or where overall drops below 0.4, gets flagged for review. Aggregate these scores over time and you have a continuous quality signal: not whether your agent is fast, but whether it is right.
Layer 3: Closed-Loop Learning
The third layer is what separates teams that keep getting better from teams that keep responding to the same incidents. Every production failure becomes a permanent evaluation case.
When your LLM judge flags a step, or a human reviewer catches a hallucination, the trace for that failure gets converted into a specific test: a task, a context, a set of expected behaviors, and a set of prohibited behaviors. That test enters your CI/CD pipeline. Before any future deployment, the agent must pass every accumulated failure case from production.
This is evaluation-driven observability. Over time, your test suite becomes a living record of every class of failure your agent has ever exhibited in the real world, and your deployments grow progressively safer.
Step-Level Failures: Where Agents Actually Break
One of the most common mistakes in agent evaluation is measuring only final task completion: did the agent solve the problem or not? This binary view hides the most actionable diagnostic information.
Recent agent research classifies failures by where in the reasoning chain they occur and what type of failure they represent. The taxonomy is consistent across evaluation frameworks:
- Tool-Skip: The agent identifies a task that requires a tool but does not call it, instead fabricating a plausible-sounding result.
- Result-Ignore: The agent calls the tool, receives a result, and then proceeds as if the result said something different.
- Output-Fabrication: The agent produces a final answer that contradicts information in its own context window.
- Parameter-Error: The agent calls the right tool with wrong or malformed parameters.
- Policy-Violation: The agent takes an action it should not be permitted to take given the user's permissions or the system's guardrails.
The distribution of these failure types varies by domain. Question-answering agents tend to fail early, with reasoning and format errors appearing in the first few steps. Tool-heavy agents in finance, law, and medicine tend to fail later: policy violations and logic errors often emerge only after several apparently correct steps have been taken.
def classify_tool_failure(trace_step: dict) -> str:
"""
Classify a failed tool interaction from a trace step.
Returns a failure mode label for root-cause analysis.
"""
tool_calls = trace_step.get("tool_calls", [])
tool_results = trace_step.get("tool_results", [])
model_output = trace_step.get("model_output", "")
task_required_tool = trace_step.get("metadata", {}).get("tool_required", False)
# Tool was needed but never called
if task_required_tool and not tool_calls:
return "TOOL_SKIP"
if not tool_calls:
return "NO_FAILURE" # Tool was not required at this step
# Tool was called but result was not referenced in output
if tool_results and not any(
str(r.get("key_value", "")) in model_output for r in tool_results
):
return "RESULT_IGNORE"
# Tool was called with missing required parameters
for call in tool_calls:
required = call.get("required_params", [])
provided = call.get("params_provided", {})
if any(p not in provided for p in required):
return "PARAMETER_ERROR"
# Tool call was flagged as outside permitted scope
for call in tool_calls:
if call.get("policy_violation", False):
return "POLICY_VIOLATION"
return "UNCLASSIFIED_FAILURE"
Tagging production trace failures with these labels lets you answer questions like: "Has our policy-violation rate increased since the last deployment?" or "Which step in our finance agent is responsible for 80 percent of Result-Ignore failures?" Without step-level classification, those questions have no answer.
Detecting Behavioral Drift Before Users Notice
Infrastructure monitoring has mature alerting: if p99 latency exceeds 2 seconds, fire an alert. Behavioral observability needs the equivalent for quality metrics. The mechanism is baseline tracking and drift detection.
Compute your agent's behavioral baseline from a representative set of production traces, ideally the first week after a stable deployment. For each metric you care about (hallucination risk, tool-selection accuracy, retrieval relevance, policy-violation rate), calculate the mean and standard deviation under normal conditions. Then monitor those metrics continuously and alert when they drift beyond acceptable bounds.
import numpy as np
from dataclasses import dataclass
@dataclass
class BehavioralBaseline:
metric_name: str
mean: float
std: float
sample_count: int
alert_threshold_std: float = 2.5 # alert if drift exceeds 2.5 std deviations
def is_drifting(self, current_value: float) -> tuple[bool, float]:
if self.std == 0:
return False, 0.0
z_score = abs((current_value - self.mean) / self.std)
return z_score > self.alert_threshold_std, z_score
def compute_baselines(
historical_scores: dict[str, list[float]],
) -> dict[str, BehavioralBaseline]:
"""
historical_scores: {metric_name: [score_1, score_2, ...]}
Returns a baseline object per metric.
"""
baselines = {}
for metric, values in historical_scores.items():
arr = np.array(values)
baselines[metric] = BehavioralBaseline(
metric_name=metric,
mean=float(arr.mean()),
std=float(arr.std()),
sample_count=len(values),
)
return baselines
def check_for_drift(
baselines: dict[str, BehavioralBaseline],
current_window: dict[str, float],
) -> list[dict]:
alerts = []
for metric, current_value in current_window.items():
if metric not in baselines:
continue
drifting, z_score = baselines[metric].is_drifting(current_value)
if drifting:
alerts.append({
"metric": metric,
"baseline_mean": baselines[metric].mean,
"current_value": current_value,
"z_score": round(z_score, 2),
"severity": "high" if z_score > 4.0 else "medium",
})
return alerts
The critical discipline here is calibrating thresholds to your domain, not to generic rules. A hallucination risk score of 0.3 might be perfectly acceptable for a creative brainstorming agent and completely unacceptable for a medical information assistant. The numbers only carry meaning in the context of what your agent is supposed to do and what the consequences of errors are.
Domain-Specific Observability: One Size Fits No One
The most frequently cited mistake in agent observability is treating it as a generic problem. You cannot install a dashboard, flip a few switches, and get meaningful quality signals across all agent types. Quality is always defined relative to the domain and the stakes.
A financial agent querying customer account data needs hard guardrails on data access scope, tight monitoring of unauthorized query patterns, and strict accuracy validation against structured data sources. A legal research agent needs citation traceability and policy-compliance checks. A medical information agent needs hallucination detection calibrated to clinical standards, not consumer NLP benchmarks.
The practical implication: define your behavioral metrics before you build your agent, not after you deploy it. For each domain, answer three questions:
- What does a correct response look like, and how would you verify it programmatically?
- What does a dangerous failure look like (not just a wrong answer, but a harmful one)?
- What behavioral patterns in the trace would distinguish the two?
The answers to those questions are your observability specification. They tell you what to log, what to score, what to alert on, and what to block. Teams that skip this step end up with generic dashboards that feel like observability but provide no signal when their agent's specific failure modes appear.
Prompt Injection: The Silent Attack Your Metrics Will Miss
Infrastructure monitoring will never catch a successful prompt injection attack. The agent completes its task, returns a response, and everything looks normal. The attack succeeded because the agent's behavior was hijacked, not its infrastructure.
Implicit prompt injection is the most dangerous variant. It arrives not in the user's direct message but in content the agent trusts: a document it was asked to summarize, a webpage it was asked to browse, a database record it was asked to interpret. Embedded in that content is an instruction that redirects the agent's subsequent tool calls.
Behavioral observability can catch this where output-based safety checks cannot. The signal is a mismatch between the user's stated task and the agent's actual tool call sequence. If a user asks an agent to summarize a PDF and the trace shows an outbound HTTP request to an unrecognized endpoint, that is not a latency anomaly. That is an incident.
Specific patterns to monitor in traces:
- Tool calls to domains not in an explicit allowlist for the agent's task type
- Tool parameters that contain strings passed directly from retrieved content without sanitization
- Sudden changes in tool-call sequence that do not match the user's original intent
- Outbound data containing PII or context window contents
These patterns require trace-level visibility. You cannot detect them from response content alone, and you certainly cannot detect them from infrastructure metrics.
The Observability Tooling Landscape
Several platforms have emerged to operationalize agent observability, each with a different center of gravity:
LangSmith provides deep integration with LangChain-based agents. Its trace UI makes it easy to navigate multi-step reasoning chains and inspect individual tool calls. Evaluation datasets and LLM-judge scoring are first-class features.
Langfuse is open-source and self-hostable, which matters for teams with data residency requirements. It supports prompt versioning alongside tracing, making it easier to correlate quality changes with prompt changes.
Braintrust focuses on the evaluation side: it is strong at running structured evals over historical traces and building regression test suites from production failures. Its CI/CD integration is among the most mature in this space.
MLflow has expanded its tracing capabilities for LLM workloads and is a natural fit for teams already using it for traditional ML experiment tracking. Unified lineage from model training through agent deployment is a genuine differentiator.
Arize Phoenix (open-source) and Galileo sit more on the analysis and alerting side: they consume traces from various sources and apply automated anomaly detection and failure classification on top.
No single tool covers the full stack. Most mature teams compose their stack: a tracing platform for instrumentation and storage, an evaluation platform for quality scoring, and custom drift-detection logic tuned to their domain. The important thing is not which tools you choose but that all three layers (tracing, evaluation, and closed-loop feedback) are present and connected.
Converting the First Incident Into a Permanent Guardrail
The single highest-leverage action you can take after your first production behavioral failure is to turn it into a test. Not a post-mortem. Not a Slack thread. A test.
Pull the full trace from the incident. Identify the step where the failure began. Write a test that presents the same task and context to your agent and asserts the absence of the specific failure mode you observed: the fabricated fact, the wrong tool call, the ignored retrieval result. Add that test to your CI pipeline.
Now that failure cannot regress silently. Every future deployment must prove it has not reintroduced the same behavior. Over time, as more incidents occur and more tests accumulate, your CI suite becomes a comprehensive behavioral specification for your agent. Deployments become safer because each one is checked against the entire history of production failures, not just the happy paths your engineers thought to test before shipping.
This is the closed-loop model in practice. It is not glamorous, but it is the mechanism by which teams with sophisticated observability continuously improve rather than perpetually fight the same fires.
Conclusion
Infrastructure monitoring is table stakes for any production system. For LLM agents, it is also the floor of a building you have not started constructing yet.
Agents that perform reliably in production over months and years share a common foundation: structured traces that capture full reasoning chains, continuous quality evaluation that scores correctness at every step, and a closed feedback loop that converts each production failure into a test that cannot silently recur.
The gap between most teams and the leading ones is not a gap in infrastructure sophistication. It is a gap in behavioral visibility. Closing it requires new tooling, new metrics, and a new mental model: an agent that returns 200 OK has not necessarily done its job correctly, and your monitoring needs to be able to tell the difference.