Stop Blaming the Model: State Coherence Is an Architecture Problem
When a multi-step AI agent workflow breaks, the instinct is to rewrite the prompt. Add more instructions, tighten the wording, or upgrade to a larger model. This approach rarely works, and the reason is straightforward: most failures in production agent systems are not reasoning failures. They are architectural failures. The agent forgot earlier context. Two agents wrote contradictory values to the same field. A hallucinated tool parameter propagated silently through four downstream steps before anyone noticed.
The takeaway is simple, even if the engineering is not: to build multi-step agent workflows that behave predictably, you have to treat state, control flow, and recovery as load-bearing structural concerns, not as afterthoughts to be patched with better prompts.
The Real Failure Mode: Silent State Corruption
Imagine a customer-service agent workflow spanning several steps: retrieve account data, classify the request, update the shipping address, confirm the change. Now imagine two sub-agents running in parallel, one pulling the current address from a cache while the other has already written the new one to the database. Neither agent errors. The workflow "succeeds." The customer receives a confirmation for the wrong address.
This is the canonical failure pattern in multi-step agent systems: not a visible crash, but a silent contradiction that compounds through downstream steps. The agents operate on stale or inconsistent views of the world, and because neither raises an exception, the corruption is invisible until something externally observable goes wrong.
The root causes are architectural. Missing state persistence means agents reconstruct context from scratch at each step. Absent consistency rules mean multiple agents can diverge on what the current state actually is. Stale reads happen when agents pull from caches that were not invalidated after a write. These are not new problems; they are the same problems database engineers solved decades ago. The difference is that LLM agent systems rarely borrow those solutions.
Control Flow Is Code, Not a Prompt
One of the most effective shifts in production agent design is deceptively simple: write control flow in code, not in natural language.
Traditional agent designs ask the LLM to decide what to do next. "Given this context, determine the next step and execute it." This works for short, well-scoped tasks under favorable conditions. It breaks under load, with long workflows, and whenever the model's reasoning is even slightly off. A model that decides to skip a validation step, retry an already-completed action, or enter an infinite clarification loop creates failures that no amount of prompt engineering reliably prevents.
Control-flow-first architecture inverts this. The program determines execution order, branching conditions, loop limits, and stop conditions. The LLM handles the parts that require natural language understanding: classifying, drafting, extracting, evaluating. Everything else is ordinary code.
def run_pipeline(topic: str, article_dir: str) -> dict:
state = load_checkpoint(article_dir) or {}
if "research" not in state:
state["research"] = researcher_agent(topic)
validate_output(state["research"], schema=ResearchSchema)
save_checkpoint(article_dir, state)
if "draft" not in state:
state["draft"] = drafter_agent(topic, state["research"])
validate_output(state["draft"], schema=DraftSchema)
save_checkpoint(article_dir, state)
if "review" not in state:
state["review"] = reviewer_agent(state["draft"])
save_checkpoint(article_dir, state)
return state
The LLM does not decide whether to run the drafter after the researcher. The code does. The LLM does not decide whether its output was valid. A schema does. This separation is not a limitation; it is what makes the system debuggable.
State as a First-Class Primitive
In most early agent experiments, state is implicit: the full conversation history is the state. This works up to a point, and then it does not, because token windows are finite, models lose coherence as context grows, and there is no recovery path if the process crashes.
Production systems promote state to an explicit layer with its own persistence strategy.
Short-term memory holds the current working context: recent tool outputs, the current step's inputs and results, and intermediate values needed in the next few steps. Redis or in-process caches work well here. This memory is fast and disposable.
Long-term memory holds summaries, past decisions, user preferences, and domain knowledge accumulated across runs. Vector databases enable semantic retrieval of relevant past context without loading everything into the prompt.
Checkpoints are the most important piece. At every meaningful boundary (task completion, a decision branch, a validation gate, a transition between agents) the system writes current state to stable storage. When step 8 fails, the workflow resumes from the last checkpoint rather than re-running steps 1 through 7.
import json, os
CHECKPOINT_FILE = "state.json"
def save_checkpoint(article_dir: str, state: dict) -> None:
path = os.path.join(article_dir, CHECKPOINT_FILE)
with open(path, "w") as f:
json.dump(state, f, indent=2)
def load_checkpoint(article_dir: str) -> dict | None:
path = os.path.join(article_dir, CHECKPOINT_FILE)
if not os.path.exists(path):
return None
with open(path) as f:
return json.load(f)
This is not novel engineering. It is the same principle as database write-ahead logging or Kubernetes pod restart policies. What is novel is how often agent systems skip it entirely and then wonder why retries are expensive and unreliable.
Prompt Chaining and Explicit State Handoff
Breaking a complex workflow into a chain of focused sub-tasks is the most direct way to reduce hallucination accumulation. When you ask a single LLM call to research a topic, write a draft, review it for accuracy, extract key claims, and format for publication, every weakness in the first step amplifies through every subsequent step.
Prompt chaining decomposes this. Each step gets a single, well-scoped prompt with controlled inputs and a validated output. The handoff between steps is explicit code, not implicit context carried in a growing conversation history.
from pydantic import BaseModel
class ResearchOutput(BaseModel):
summary: str
key_points: list[str]
sources: list[str]
def researcher_agent(topic: str) -> ResearchOutput:
raw = call_llm(system_prompt=RESEARCHER_PROMPT, user_message=topic)
# Parse and validate before handing off
return ResearchOutput.model_validate_json(raw)
def drafter_agent(topic: str, research: ResearchOutput) -> str:
# Only pass what the drafter needs, not the full conversation history
context = f"Topic: {topic}\n\nKey points:\n" + "\n".join(
f"- {p}" for p in research.key_points
)
return call_llm(system_prompt=DRAFTER_PROMPT, user_message=context)
Two details matter here. First, the researcher's output is validated against a schema before it goes anywhere. A malformed or hallucinated output raises an exception at the boundary, not three steps later. Second, the drafter receives a carefully scoped context rather than the entire research conversation. Selectively propagating only relevant outputs keeps context within reliable limits and prevents earlier reasoning tangents from contaminating later steps.
Validation Checkpoints and Schema Enforcement
Tool divergence is one of the most common failure modes in agentic systems: the LLM produces a tool call with a hallucinated parameter name, an out-of-range value, or a structurally malformed payload. Without interception, that bad call propagates downstream. Depending on the system, it either errors loudly (recoverable) or silently writes garbage to a database (not recoverable without a detailed audit trail).
Schema enforcement at every boundary converts silent failures into loud, catchable exceptions. Pydantic models, structured output constraints from the model provider, or JSON Schema validators all work. The choice matters less than the discipline of always validating before passing data to the next step.
from pydantic import BaseModel, ValidationError
class ToolCallParams(BaseModel):
action: str
entity_id: str
new_value: str
def safe_tool_call(raw_params: dict) -> ToolCallParams:
try:
return ToolCallParams.model_validate(raw_params)
except ValidationError as e:
# Log, checkpoint, and raise -- do not swallow silently
log_validation_failure(raw_params, e)
raise
End-state evaluation is equally important. Rather than validating every intermediate step in exhaustive detail, focus validation energy on the transitions that matter: the output of each major stage before it feeds the next, and the final state of the workflow before any external write. The question to answer at each gate is not "did the LLM reason correctly?" but "is this output structurally valid and in-range before we hand it off?"
Recovery and Fallback Strategies
When a step fails, the question is not whether to retry. It is how to retry, and how to degrade gracefully when retries are exhausted.
Effective systems separate the orchestration layer (which handles timeouts, retry policies, and circuit breakers) from the reasoning layer (the LLM's problem-solving). The orchestrator does not need to understand why a step failed. It needs to know what safe restore points exist, how many attempts have been made, and which fallback path to take.
A tiered fallback approach looks roughly like this:
- Standard retry: Re-run the failed step with the same parameters after a short delay.
- Adjusted retry: Re-run with reduced temperature or a more constrained prompt to tighten the model's output.
- Smaller model with tighter constraints: Swap to a faster, cheaper model with a highly structured output format, sacrificing creativity for reliability.
- Human escalation: Surface the failure to a human reviewer with enough context to understand what went wrong and where the workflow stopped.
The checkpoint pattern from the previous section is what makes tiered recovery safe. Each retry can reload from the last valid checkpoint rather than re-executing the entire workflow. Without checkpoints, aggressive retries waste compute and risk thrashing through the same failure repeatedly.
Circuit breakers prevent cascade failures. If an external tool (a search API, a database, a third-party service) starts returning errors, a circuit breaker opens and prevents the agent from hammering the degraded dependency. This is standard distributed systems engineering applied to agent orchestration.
Multi-Agent Coordination: Scoped Visibility and Synchronization
When multiple agents share access to the same state (a user record, a document, a task queue), correctness requires more than good intentions. It requires explicit protocols.
Four properties determine whether a multi-agent system handles shared state correctly.
Scoped visibility: Each agent can only read and write the state it is explicitly authorized to touch. An agent handling address updates should not be able to read payment information. An agent summarizing a document should not be able to modify it.
Policy enforcement: Write permissions are checked at the state layer, not assumed based on which agent is making the request.
Temporal correctness: When multiple agents read a record at the same time, they all see the same version. Stale reads that diverge mid-workflow are the root cause of the shipping address problem described at the start of this article.
Provenance: Every state change carries metadata (which agent made it, when, and from what prior state). This makes post-mortem debugging tractable.
A pattern borrowed directly from distributed systems design, the Saga pattern, applies compensating transactions to multi-agent planning. If agent B's step fails after agent A's step succeeded, the system executes a rollback action that undoes A's change rather than leaving the state partially updated. This brings ACID-like guarantees to LLM workflows that previously had none.
The alternative, having agents periodically debate or self-check for consistency, does not work reliably. Standard self-consistency methods cannot surface cross-agent state mismatches because each agent's internal reasoning is coherent within its own context. The inconsistency lives in the gap between them, and dedicated synchronization protocols are the only way to close it.
Observability Built for Agents
Standard ML metrics don't measure what you actually need to know about a running agent system. Accuracy and F1 scores tell you nothing about whether the agent maintained state correctly across a twelve-step workflow, handled a timeout gracefully on step seven, or silently produced a stale read on step three.
Agentic observability requires a different set of signals:
- Checkpoint coverage: Did the workflow save state at every expected boundary?
- Validation pass rate: What percentage of stage transitions passed schema validation on the first attempt?
- Recovery rate and depth: How often did fallback tiers activate, and which tier resolved the failure?
- State divergence events: Were any cross-agent reads detected on different versions of the same record?
- Latency and cost per stage: Where is time and money being spent, and are those investments producing higher-quality outputs?
End-to-end tracing, which became broadly available in agent frameworks around 2024 and 2025, enables post-mortem analysis of specific workflow runs: which steps executed, in what order, what state was read and written, and where the failure cascade started. Without tracing, debugging a multi-step failure is essentially archaeology.
The shift in monitoring philosophy is from "did the model produce a good output?" to "did the system behave correctly?" Those are related questions, but they are not the same question, and conflating them is what leaves teams debugging prompt wording when the actual problem is a missing checkpoint.
Conclusion
Better prompts are not the solution to broken multi-step agent workflows. Control flow, state persistence, validation, recovery, and coordination are the solution. Each of these concerns has well-established engineering patterns behind it: checkpointing from database systems, schema enforcement from API design, circuit breakers from distributed systems, scoped visibility from access control. The work of building reliable agentic systems is largely the work of applying these patterns to LLM orchestration.
The practical starting point is not complex. Add explicit checkpoints at every stage boundary. Validate every output against a schema before it crosses into the next step. Write branching and retry logic in code, not in prompts. Scope what each agent can read and write. Add tracing so you can reconstruct what actually happened when something goes wrong.
When LLM reasoning inevitably breaks down (and it will), these structural controls determine whether the failure is a recoverable, auditable event or a silent corruption that surfaces three workflows later in a customer complaint. That difference is entirely in the architecture.