The Real Cost of Running Agents in Production
Token math, infrastructure overhead, and where the money actually goes.
The number nobody wants to calculate
Builders love to talk about what their agents can do. Nobody talks about what they cost per request.
Here's the uncomfortable math: a multi-agent system that uses Claude Opus for reasoning, Sonnet for drafting, and calls 5 tools with context injection can cost $0.50-3.00 per run. At 1,000 runs per day, that's $500-3,000/day in API costs alone — before infrastructure, observability, and engineering time.
Most startups don't discover this until their first real-traffic week.
Where the money goes
LLM API costs (60-80% of total)
The dominant cost. Driven by three factors:
- Model choice. Opus costs ~15x more than Haiku per token. Most builders default to the most capable model when a cheaper one would suffice for 80% of tasks.
- Context length. Every tool result, every intermediate reasoning step, every piece of retrieved context goes into the prompt. A RAG pipeline that stuffs 10 document chunks into context is paying for 10x the input tokens.
- Retry and re-prompting. When the LLM produces invalid output and you re-prompt with the error, you're paying for the entire context again plus the error message.
Infrastructure (10-20%)
- Vector database hosting (Pinecone, Weaviate, etc.)
- MCP server hosting (if remote)
- Queue infrastructure for async tasks
- Storage for agent state and checkpoints
Observability (5-10%)
- Trace storage (Langfuse, LangSmith, Arize)
- Log aggregation
- Custom dashboards
Hidden costs (often ignored)
- Developer time debugging. Non-deterministic systems take 3-5x longer to debug than deterministic ones.
- Prompt engineering iterations. Each iteration means more test runs, more tokens, more cost.
- Retry storms. A single tool failure can trigger cascading retries across a multi-agent system.
Optimization strategies that actually work
1. Model routing
Not every task needs your most expensive model. Build a router:
- Classification and extraction → Haiku ($0.25/1M input tokens)
- Drafting and summarization → Sonnet ($3/1M input tokens)
- Complex reasoning and planning → Opus ($15/1M input tokens)
A well-designed router can cut LLM costs by 60-70% with negligible quality loss. The router itself can be a Haiku call that reads the task and picks a model.
2. Context window management
Every token in your context costs money. Be aggressive about what goes in:
- Summarize tool results before injecting them. A 5,000 token API response can usually be condensed to 200 tokens without losing the information the agent needs.
- Use sliding windows for long conversations. The agent doesn't need the full history — it needs the last N relevant messages.
- Cache frequently used context. If every agent run includes the same system prompt and reference data, cache the KV pairs (Anthropic's prompt caching reduces costs by up to 90% for cached prefixes).
3. Caching at every layer
- Semantic caching: If a user asks the same question twice, return the cached answer. Tools like GPTCache and Redis with vector similarity make this practical.
- Tool result caching: If your agent calls the same API with the same parameters, cache the result. Set TTLs based on data freshness requirements.
- LLM response caching: For deterministic tasks (classification, extraction), cache the model's output for identical inputs.
4. Token budgets
Set hard limits per agent, per run, and per tool invocation. When a budget is exhausted:
- Stop the current task and return partial results
- Switch to a cheaper model
- Fall back to a template response
Without budgets, a single confused agent can burn through your daily allocation in minutes.
5. Batch processing
If your use case allows it, batch requests instead of processing them individually. Anthropic's Message Batches API gives you 50% off for async processing. For overnight jobs (scoring pipelines, data enrichment, content generation), this is free money.
Cost monitoring
You cannot optimize what you don't measure. At minimum, track:
- Cost per agent run (broken down by model, tools, retries)
- Cost per user request (end-to-end, including all agent interactions)
- Cost per tool invocation
- Retry rate and retry cost
- Cache hit rate
Langfuse and Helicone both track per-request costs. Arize Phoenix gives you cost breakdowns in trace views. If you're not using any of these, you're flying blind.
The math that matters
Before you build, estimate:
- Average tokens per run (input + output)
- Expected daily volume
- Retry rate (assume 15% to start)
- Model mix (what % needs the expensive model?)
Multiply it out. If the number is uncomfortable, redesign before you build. Switching from Opus to Sonnet after launch is a product change, not an optimization.
Bottom line
Agents are expensive. The builders who succeed are the ones who treat cost as a first-class design constraint, not an afterthought. Budget your tokens like you budget your engineering hours — because at scale, they cost about the same.