TL;DR: Most LLM agent cost optimization advice targets model selection, prompt caching, and context compression. All three assume the model must be invoked. In one document-processing workflow we instrumented internally, compiling deterministic decision branches into plain code eliminated LLM calls across most execution paths and cut token spend by more than 80% before any of those standard levers were applied. The harness sets the cost floor. Everything else scales within it.
Key Takeaways
- The harness beats the model: Redesigning how your agent makes decisions moves the needle far more than swapping to a cheaper model.
- Repeatable decisions are waste: Any decision your agent makes the same way every time should be replaced with plain code, cutting those tokens to zero.
- Agents multiply your token bill fast: Unlike chatbots, agents loop through many decisions per task, so inefficiencies compound quickly.
- Audit before you optimize: Instrument your workflow first. You cannot know which decisions to compile out until you see where tokens are actually spent.
- Compilation is not caching: Prompt caching speeds up repeated LLM calls. Compiling a decision into code eliminates the call entirely.
- Remaining calls improve too: Once deterministic branches are removed, the surviving LLM calls receive cleaner, more focused context and become better candidates for routing and caching.
Introduction
AI agents generate 10x to 100x more tokens than chatbots, which makes inference a real cost problem for enterprise teams. Every published guide on LLM agent cost optimization converges on three levers: model routing, prompt caching, and context compression. Those levers combined can reach 70-85% API spend reduction, but all three share a silent assumption: the LLM must be invoked.
In one document-processing workflow we instrumented internally, a team found a lever upstream of all three. They audited their harness, identified which decisions had converged into rules, and compiled those branches into plain code. Token spend dropped more than 80% as a result of that single structural change, before touching the model, tuning a prompt, or configuring a cache. The figure is drawn from that internal instrumentation and is disclosed here as proprietary author research, not an independently published study.
Why does model switching alone rarely fix an AI agent's cost problem?
Model switching rarely fixes cost because the harness sets the baseline, and the model only multiplies whatever inefficiency is already built in.
If your agent runs 12 LLM calls per task and 8 always produce the same output, model routing's 40-70% savings apply to all 12. But those 8 deterministic calls should cost zero, not a reduced rate. You are cutting the price on calls that should not exist.
The Cost Surface Stack is a practical hierarchy used in this guide to make the audit order explicit. It has three layers:
| Layer | What It Controls | Typical Savings Ceiling | Audit Order |
|---|---|---|---|
| Harness design | Whether the LLM is invoked at all | 100% on eliminated branches | First |
| Model choice | Price and quality per invocation | 40-70% on routed calls | Second |
| Prompt engineering | Token count per call | Varies by context length | Third |
Most teams audit in the wrong order, starting at prompt engineering or model selection and never reaching the cost floor. Applying model routing to a bloated harness is optimizing the wrong variable entirely.
What is decision compilation, and how is it different from prompt caching?
Decision compilation replaces convergent agent branches with plain code, permanently eliminating those LLM calls rather than making them cheaper.
Caching stores the model's response and serves it again. The inference path still exists, and a cache miss still triggers full inference. Compilation removes the call from execution entirely. A document triage agent that routes PDFs under two pages to a short-form handler with near-total consistency across a large run sample is not making a judgment. It is following a rule: if len(pages) < 3: route_to("short_form"). Token cost for that branch: zero.
| Technique | LLM Still Invoked? | Savings Type | One-Time or Ongoing? | Typical Savings |
|---|---|---|---|---|
| Prompt caching | Yes (on cache miss) | Per-call cost reduction | Ongoing tuning | Varies by repetition rate |
| Model routing | Yes (cheaper model) | Per-token price reduction | Ongoing maintenance | 40-70% on routed calls |
| Context compression | Yes | Fewer tokens per call | Ongoing tuning | Varies by context length |
| Decision compilation | No | Eliminates call entirely | One-time audit + refactor | 100% on compiled branches |
How do you audit an agent harness to find decisions ready to compile?
Instrument every decision node, log input conditions and output branches, then look for branches where model response variance is near zero. Those branches have already become rules.
The ICVR Loop: A Four-Step Audit Method
- Instrument every decision node to emit: input features, branch taken, output token count, and task outcome. You cannot identify deterministic agent branches by reading code. You find them by logging.
- Converge by analyzing logs for branches where model output matches the same result the overwhelming majority of the time across diverse inputs. Low variance is your decision loop refactoring candidate list.
- Validate before removing any call. Inspect a representative sample of the candidate branch, confirm the rule is input-driven, and run the deterministic equivalent in shadow mode alongside the model for a sustained period before cutting over.
- Remove the inference call once shadow mode confirms parity. Log the removal date and rule version so you can catch drift if data distribution shifts later.
What did the 80%+ token reduction actually look like in practice?
Compiling seven deterministic decision branches into plain code eliminated LLM calls across most execution paths in one internally instrumented workflow, cutting token spend by more than 80% with no degradation in output quality.
This figure is the result of proprietary author research on a single production document-processing workflow and should be read as a case study rather than a generalizable benchmark. Instrumentation ran across a large production sample and revealed that most decision nodes showed near-zero output variance.
The compiled branches covered: file-type routing, length classification, language detection, formatting normalization flagging, duplicate detection, priority-tier assignment, and output destination routing. The remaining nodes, including semantic classification of ambiguous content, extraction from unstructured fields, quality scoring, and exception handling, stayed LLM-powered.
Combined optimization levers are documented at 50-80% savings for enterprise teams. The internally instrumented workflow reached the upper bound of that range through a single structural change before any of those secondary levers were applied. The surviving LLM calls also improved: they received cleaner, more focused context, making them stronger candidates for routing and caching in a subsequent optimization pass.
Frequently Asked Questions
Is decision compilation only useful for large-scale agent deployments? The savings become most material at scale. Enough daily task volume makes even one compiled branch meaningfully cost-reducing. Below that threshold, instrument first and compile when scale warrants it. The audit itself costs very little and surfaces useful data regardless of whether you compile immediately.
How do you handle compiled branches that occasionally need to revert to LLM judgment for edge cases? Build a fallback: the deterministic rule runs first, and inputs outside its validated range route to the LLM. If fallback frequency climbs beyond a threshold you define for your workflow, the rule needs refinement or that branch is not ready to compile.
Does decision compilation conflict with prompt caching strategies? They work together. Compile deterministic branches first to zero out that token spend, then apply caching to the remaining LLM-dependent branches. The two techniques target different parts of the cost surface and do not interfere.
How do you prevent compiled rules from drifting as data distribution changes? Version every rule with an activation date and input-feature fingerprint, then run periodic shadow-mode audits replaying recent production inputs through both the rule and the live model. Flag divergence above your chosen threshold for revalidation and treat it as a prompt to re-enter the ICVR Loop at the Validate step.
Conclusion
Instrument one agent workflow's decision nodes and log input conditions and output branches over a meaningful production window. If any branch shows near-total output consistency, you have a compilation candidate and a permanent cost reduction waiting in a small amount of targeted code.
The Cost Surface Stack and the ICVR Loop, both author synthesis frameworks presented in this guide, give you a repeatable method to find that candidate and act on it without guesswork.
The cheapest LLM call is the one you never make.
-> Explore our guides on agent instrumentation patterns and harness architecture before the audit phase begins.
Learn from me
Agent Engineering Bootcamp: Developers Edition, my Maven cohort. Advanced agentic RAG, multi-agent orchestration, memory, evals, and guardrails. Take agents from prototype to production. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
