LLM Cost Optimization: Cutting Token Spend Without Hurting Quality

LLM costs start small and then surprise you. A prototype costs a few dollars a day; the same feature at production volume, with longer prompts, retries and agent loops, becomes a meaningful line item. The good news is that most LLM spend is waste you can engineer away without users noticing any drop in quality.

#Key takeaways

• Measure cost per successful task, not cost per token.
• The biggest savings usually come from model routing (smaller models for easy work) and prompt caching.
• Output tokens often cost several times more than input tokens, so control verbosity.
• Use batch APIs for anything that isn't interactive.
• Protect quality with evals: every optimization should be tested, not assumed.

#First: understand where the money goes

Pricing is per token, usually split into input and output, and output is typically priced higher. Your bill is roughly:

textcost ≈ Σ requests × (input_tokens × input_price + output_tokens × output_price)

Before optimizing, instrument every call with:

• feature / endpoint name
• model and version
• input, cached-input and output token counts
• latency and success / failure
• a task ID tying multiple calls together (important for agents)

Then look at the distribution. Typically a few features and a few call patterns, such as a bloated system prompt or an agent loop that runs 15 steps, account for most of the spend.

#1. Route requests to the right-sized model

Not every request needs your most capable model. Classification, extraction, short rewrites and simple Q&A often work just as well on smaller, much cheaper models.

Routing strategies:

• Static routing by task: the summarizer uses a small model, complex analysis uses a large one.
• Cascade: try the small model first; escalate if confidence is low or validation fails.
• Classifier router: a cheap model or heuristic decides the difficulty tier up front.

Validate each route with your eval set. Our LLM evaluation guide covers how.

#2. Use prompt caching

Many providers discount input tokens that repeat a previously seen prefix: system prompts, tool definitions, reference documents, few-shot examples. Savings on cached tokens can be large, and latency drops too.

To benefit:

• Put stable content first and variable content (user input) last.
• Don't inject timestamps or request IDs into the top of the prompt, because they break the cache.
• Keep tool definitions and system prompts byte-identical across requests.

#3. Trim your prompts

Prompts grow by accretion: one more rule, one more example, one more paragraph of context. Audit them:

• Remove instructions the model follows anyway (test by removing and re-running evals).
• Shorten few-shot examples, or reduce from five to two.
• In RAG, send fewer, better chunks after reranking rather than everything that matched.
• Summarize long conversation history instead of resending it verbatim.

#4. Control output length

Output tokens are expensive and slow. Techniques:

• Set max output tokens per use case.
• Ask for concise formats: "Return only the JSON", "Max 3 bullet points".
• Use structured outputs for machine consumers. No preamble, no "Certainly! Here's…".
• For reasoning models, choose the lowest reasoning effort that passes your evals.

#5. Batch non-interactive work

Nightly enrichment, bulk classification, document backfills and eval runs don't need real-time responses. Many providers offer batch APIs at a significant discount in exchange for asynchronous completion. If a user isn't waiting, batch it.

#6. Cache responses

• Exact-match caching: identical inputs return stored outputs. Great for repeated queries, deterministic transformations and eval re-runs.
• Semantic caching: embed the query and return a stored answer when a very similar query was answered recently. Useful for FAQ-style traffic, but set a strict similarity threshold and never cache personalized or permissioned answers across users.

#7. Fix agent loops

Agents can burn tokens fast: every step resends the growing context. Guardrails:

• Cap steps and total tokens per task.
• Trim tool outputs: return the 5 fields the agent needs, not a 20 KB JSON blob.
• Summarize working memory periodically.
• Detect loops, such as the same tool called with the same arguments.

We cover this in depth in making AI agents reliable.

#8. Reduce retries and failures

Every failed call you retry costs twice. Common causes: invalid JSON (fix with schema-constrained outputs), timeouts on overly long generations (cap output), and rate-limit errors (use backoff with jitter and client-side rate limiting).

#9. Consider fine-tuning or distillation at scale

At high volume on a narrow task, fine-tuning a smaller model on a larger model's validated outputs can cut per-request cost substantially while keeping quality. Worth it once volume is high and the task is stable. See fine-tuning vs RAG vs prompting.

#10. Self-host when the math works

Running open-weight models on your own GPUs can be cheaper at sustained high utilization, and it's sometimes required for data residency. But include the full cost: GPUs sitting idle, inference-server engineering, scaling, monitoring and upgrades. For spiky or low-volume workloads, APIs usually win.

#Optimization priority table

#Build a cost dashboard

Make cost visible to the people who can change it:

• Cost per feature, per day, per model.
• Cost per successful task (for example, cost per resolved support ticket).
• Top 20 most expensive requests, which often reveal bugs like runaway loops or giant context.
• Alerts on anomalies, such as daily spend 2× the trailing average.

#FAQ

#Why are output tokens more expensive than input tokens?

Input tokens are processed in parallel during the prefill phase. Output tokens are generated one at a time, which uses GPU capacity less efficiently. Providers price that difference in.

#Does a smaller model always mean lower quality?

No. For well-defined tasks like classification, extraction and formatting, smaller models often match larger ones. The only way to know is to run both against your eval set.

#Is semantic caching safe?

It can be, for non-personalized and non-sensitive queries with a high similarity threshold. It's risky when answers depend on user identity, permissions or time. Scope caches carefully and monitor for stale or wrong hits.

#How do I estimate costs before launch?

Multiply expected requests by average input and output tokens, measured on realistic test traffic, not guesses. Add overhead for retries, agent steps and evals. Then load-test with real prompts to confirm.