Sign in Subscribe
By Marvin —

How Context Bloat Destroys Agent Economics (And the Playbook to Fix It)

If your AI agent was profitable at 1,000 requests per month but bankrupt at 10,000, context bloat is probably the reason. Every turn of a multi-turn session appends more tokens to the next call, and most teams never instrument that growth until the invoice arrives. The good news: the same workload that bleeds money under naive context management can run profitably with a handful of targeted strategies. This article explains why context grows out of control, what it actually costs, and how to compress multi-turn sessions so cost scales sublinearly with usage.

The Linearity Trap

Most agent frameworks handle context the same way: every turn, append the new user message and assistant response to the existing history, then send the whole thing to the model. It is simple, correct, and economically catastrophic at scale.

The math is unforgiving. A 20-turn conversation consumes 5,000 to 10,000 tokens when only 500 to 1,000 tokens of genuinely useful context would suffice. Naive memory injection scales even worse: 24 stored entries cost roughly 594 tokens per call, but 500 entries cost around 8,000 tokens. Production agents running continuously for weeks routinely accumulate 80,000 to 120,000 token contexts. At that point the context window is not a feature; it is a liability.

The core problem is that token cost scales with the length of the entire prompt on every single call. A 100-turn session costs roughly 5,050 times the cost of a single turn just from re-reading prior context (1 + 2 + 3 + ... + 100 = 5,050), even if none of that history is relevant to the current question. Research consistently finds that 70 to 85 percent of accumulated tokens represent context the model never needed.

Where the Money Actually Goes

Before reaching for solutions, it helps to understand the cost structure of a typical production agent.

LLM API calls account for 70 to 85 percent of total AI agent operating costs. Within those calls, re-sent context (system prompts, tool definitions, and state history) accounts for roughly 62 percent of the total inference bill. That is the largest single cost category, and it is almost entirely invisible without deliberate instrumentation.

Output costs amplify the problem. The median output-to-input cost ratio at frontier models sits around 4:1, meaning a verbose response compounds any waste already baked into the input. And premium pricing adds another multiplier: requests exceeding 200,000 tokens cost 2x for input and 1.5x for output on several major platforms.

A concrete example shows how fast this compounds. Suppose your agent handles 10,000 sessions per month, each averaging 20 turns, with a naive full-history context of 8,000 tokens per call. At $3.00 per million input tokens, that is:

  • 10,000 sessions x 20 turns x 8,000 tokens = 1.6 billion input tokens per month
  • Cost: $4,800 per month in input tokens alone

Apply the strategies in this article and get context down to 1,500 tokens per call:

  • 10,000 sessions x 20 turns x 1,500 tokens = 300 million input tokens per month
  • Cost: $900 per month

Same quality. Same scale. One-fifth the cost.

Strategy 1: Replace Full-History Injection with Retrieval-Based Memory

The most impactful single change most teams can make is replacing full conversation history injection with semantic retrieval. Instead of re-sending every prior turn, you index past turns into a vector store and retrieve only the few that are actually relevant to the current message.

The numbers are striking. Switching from naive full-context injection to retrieval-based memory reduces token consumption from around 594 tokens per call to 166 tokens (a 72 percent reduction) with identical answer quality in benchmarks.

The implementation pattern is straightforward:

from anthropic import Anthropic
import numpy as np

client = Anthropic()

class RetrievalContext:
    """Replace full-history injection with semantic retrieval."""

    def __init__(self, embed_fn, top_k=5):
        self.embed_fn = embed_fn   # any embedding function
        self.turns = []            # list of {"role", "content", "embedding"}
        self.top_k = top_k

    def add_turn(self, role: str, content: str):
        embedding = self.embed_fn(content)
        self.turns.append({"role": role, "content": content, "embedding": embedding})

    def retrieve(self, query: str) -> list[dict]:
        if not self.turns:
            return []
        query_vec = self.embed_fn(query)
        scores = [
            np.dot(query_vec, t["embedding"]) for t in self.turns
        ]
        top_indices = np.argsort(scores)[-self.top_k:][::-1]
        return [
            {"role": self.turns[i]["role"], "content": self.turns[i]["content"]}
            for i in top_indices
        ]

    def build_messages(self, user_message: str) -> list[dict]:
        relevant_history = self.retrieve(user_message)
        return relevant_history + [{"role": "user", "content": user_message}]

The build_messages call sends only the most relevant prior turns rather than the entire history. As sessions grow to hundreds of turns, token cost per call stays roughly constant instead of growing linearly.

For teams not ready to stand up a vector store, a simpler intermediate step is token-based windowing: count the tokens in the full history, and once you exceed a threshold, drop the oldest turns first. This is better than message-count truncation because it responds to actual token consumption rather than arbitrary turn counts.

import anthropic

def trim_to_token_budget(messages: list[dict], budget: int, model: str) -> list[dict]:
    """Drop oldest turns until the message list fits within the token budget."""
    client = anthropic.Anthropic()
    while messages:
        response = client.messages.count_tokens(
            model=model,
            messages=messages,
        )
        if response.input_tokens <= budget:
            return messages
        # Drop the oldest turn (preserving system-level turns if any)
        messages = messages[1:]
    return messages

Strategy 2: Summarize and Distill Instead of Truncate

Truncation preserves recency but discards everything older. Summarization preserves semantics at a fraction of the token cost.

The practical approach is hierarchical summarization: every N turns, call the model to compress the oldest segment of the conversation into a short, structured summary, then replace those turns with the summary. The active window stays manageable; critical information survives.

Structured distillation takes this further by extracting key facts into a typed JSON object rather than free-text prose. Benchmarks show significant token reduction (in some studies exceeding 10x) while preserving retrieval quality, because the model sees dense, structured signal rather than conversational filler.

import json
import anthropic

client = anthropic.Anthropic()

def distill_turns(turns: list[dict], model: str = "claude-haiku-4-5") -> dict:
    """
    Compress a sequence of conversation turns into a structured fact object.
    Returns a dict to splice into the next prompt instead of re-sending the turns.
    """
    history_text = "\n".join(
        f"{t['role'].upper()}: {t['content']}" for t in turns
    )
    prompt = (
        "Extract the key facts, decisions, and user preferences from this conversation. "
        "Respond with a JSON object only. No prose.\n\n" + history_text
    )
    response = client.messages.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(response.content[0].text)

# Usage: after every 10 turns, distill the oldest 8 into a fact object.
# Then prepend the fact object as a system note in the next call:
#   "Context from earlier: {json.dumps(distilled_facts)}"

This pattern is especially effective for long-running assistants where the user's preferences, goals, and prior decisions matter but the specific wording of earlier messages does not.

Strategy 3: Structure Your Prompt for Maximum Cache Hits

Prompt caching is the highest-leverage optimization available on most major platforms today. On Claude, cached tokens cost 10 percent of the standard price, and the cache applies automatically to any repeated prefix pattern. A well-structured prompt can cut input costs by up to 80 percent on long-horizon agentic workloads, with zero change to output quality.

The key insight is to treat prompt structure the same way Docker treats image layers: put the most stable content first, and let the cache accumulate on it. Volatile content (the user's current message, retrieved context) goes at the end.

import anthropic

client = anthropic.Anthropic()

# Static layers: system prompt and tool list never change between calls.
# These hit the cache on every turn after the first.
SYSTEM_PROMPT = """You are a helpful technical assistant..."""  # long, stable

TOOLS = [
    # full tool definitions here
]

def call_agent(conversation_history: list[dict], user_message: str):
    messages = conversation_history + [{"role": "user", "content": user_message}]

    return client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=SYSTEM_PROMPT,   # stable prefix, cached after first call
        tools=TOOLS,            # stable prefix, also cached
        messages=messages,      # volatile suffix, not cached, but small
    )

The structural rule is: stable content first, volatile content last. If you shuffle the system prompt or vary the tool list between calls, you break the cache prefix and pay full price every time. The single most common accidental cache-buster is injecting a timestamp or request ID into the system prompt.

Strategy 4: Route to the Right Model

Not every task in an agent pipeline deserves a frontier model. A task routed to a large reasoning model can cost two orders of magnitude more than the same task routed to a fast, smaller model. Most production pipelines contain a mix of complex reasoning tasks and simple operations: reformatting output, classifying intent, checking whether a condition is met. The latter category does not need frontier capability.

Practical routing works on two signals: estimated complexity and estimated context length. Short, well-defined tasks with small contexts route to cheaper models. Tasks that require multi-step reasoning, nuanced judgment, or very long context route to frontier models.

import anthropic

client = anthropic.Anthropic()

def route_call(messages: list[dict], token_count: int, is_complex: bool) -> str:
    """Return the appropriate model name given task characteristics."""
    if not is_complex and token_count < 4_000:
        return "claude-haiku-4-5"      # fast, cheap, accurate for simple tasks
    elif token_count < 32_000:
        return "claude-sonnet-4-6"     # balanced for most agentic work
    else:
        return "claude-opus-4-5"       # reserve for genuine complexity at scale

# Before every LLM call, count tokens and classify the task,
# then pass the result to route_call to select the model.

Combining routing with context pruning produces compounding savings: trim irrelevant history before the call, then route to the cheapest model that can handle the pruned context.

Strategy 5: Manage Tools as a Cost Center

Every tool definition sent to the model gets tokenized and billed, whether or not the tool is used. A typical production agent with 20 tool definitions might spend 2,000 to 4,000 tokens per call just on tool schemas. Over thousands of calls, that adds up to a meaningful line item.

The fix is dynamic tool filtering: before each call, select only the tools relevant to the current task. A task in the "data retrieval" phase does not need write tools. A task in the "user communication" phase does not need analytics tools.

Equally important is constraining tool output. Observation masking (stripping or summarizing the raw output of a tool before appending it to context) can cut tool output from 80,000 tokens to 2,000. An API that returns a 500-field JSON object rarely needs all 500 fields to answer the current question. Extract only what is relevant, and append that to context instead.

Structured output schemas (JSON mode on the model response) prevent a related problem: verbose free-text responses that balloon output token counts. Asking the model to respond in a typed schema both constrains output length and makes the response easier to parse downstream.

Strategy 6: Design a Context Lifecycle

The strategies above work best when they are part of a deliberate context lifecycle rather than isolated optimizations bolted onto an existing system. A well-designed lifecycle treats context as a first-class resource with four phases.

Ingestion is what gets stored. Every tool output, every user turn, every assistant response is a candidate for inclusion. Filtering at ingestion (before anything hits the context) is the cheapest intervention of all.

Compression is how you trim without losing signal. This covers hierarchical summarization, structured distillation, and prompt compression tools like LongLLMLingua, which can achieve around 4x token reduction with minimal quality degradation. Only 5 to 10 percent of tokens in a typical prompt carry meaningful signal; the rest is structural noise and redundant instruction.

Retrieval is what subset you send. Retrieval-based memory, topic-gated context, and dynamic tool filtering all live here.

Refresh is when you prune. Rather than letting context grow until it hits a hard limit, a refresh strategy proactively clears old dialogue turns on a schedule or when total context exceeds a threshold, replacing them with a compact distilled summary. The agent maintains continuity without re-reading history verbatim.

The Compounding Effect

These strategies do not operate in isolation. Applied together, they compound.

Consider a baseline agent spending $4,800 per month as described earlier. Apply retrieval-based memory (72 percent reduction in context tokens), structure the prompt for cache hits (an additional 60 percent reduction on cached portions), add model routing to send simple tasks to a cheaper model (perhaps 40 percent of calls), and filter tool definitions per call. The resulting system routinely achieves 80 percent or more cost reduction versus the naive baseline, while answer quality remains high because the model is receiving the most relevant context rather than an undifferentiated dump of history.

That is the definition of sublinear growth: usage doubles, cost barely moves.

Conclusion

Context bloat is not an edge case or a scaling problem for some future version of your system. It is the default trajectory for any agent that appends conversation history without a compression strategy. The economics are hostile at frontier model prices, and they only get worse as sessions grow longer and usage scales up.

The good news is that the solutions are practical, available today, and composable. Retrieval-based memory, structured distillation, prompt caching, model routing, and tool filtering are engineering patterns that production teams are shipping right now. Each one moves cost growth in the right direction, and together they can turn a system that was economically unsustainable at scale into one that gets cheaper per useful unit of work as it grows.

The worst time to think about context economics is when the invoice arrives. The best time is before you write the first agent loop.

Subscribe to Marvin G6 | Engineering, AI & Automation

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe