LLM Tokens Explained: Cost, Thinking, Caching, Batch

A token is the small chunk of text a language model reads and writes, usually a word fragment of a few characters. You pay per token, and output tokens cost several times more than input tokens. Your bill is driven by four things: how much you send (input), how much the model writes (output), how much it silently thinks (also billed as output), and how much history you resend because the model remembers nothing between calls.

This guide explains each part and the levers that cut cost: prompt caching, batching and picking the right model. Prices below use Anthropic's Claude models as the worked example, as listed in October 2026. Other providers structure pricing the same way but with different numbers, so always check the current pricing page before budgeting.

#Key takeaways

• Tokens are the unit of cost and of context. Input, output and thinking all count.
• Output costs about 5x input on the Claude models listed below.
• Thinking tokens are billed as output, even when you never see them.
• LLMs have no memory. The API is stateless, so you resend the whole conversation every turn and pay for it again.
• Caching and batching are the two big free wins. Cache reads cost a fraction of normal input, and batch jobs get 50% off.
• Match the model to the task. The cheapest model that passes your evaluation is the right one.

#What is a token?

Models do not read letters or words. A tokenizer splits text into pieces, and each piece maps to a number the model processes. In English a token is often a short word or part of one, roughly three to four characters, though code, numbers and non-English text usually break into more tokens per character.

Token counts are specific to each model's tokenizer. Do not estimate with another vendor's tokenizer: Anthropic's guidance is that OpenAI's tiktoken undercounts Claude tokens by roughly 15 to 20% on typical text and by more on code or non-English input. To get an exact number, call the model provider's token-counting endpoint (messages.count_tokens for Claude) with the same model you will use.

#Input tokens vs output tokens

Input tokens are everything you send: the system prompt, tool definitions, the conversation history, documents and your new question. Output tokens are what the model generates.

Haiku 5.5's price applies to prompts up to 100K tokens, with higher rates beyond that. Check the live pricing page for the current rates.

#A worked example

A request with 10,000 input tokens and 1,000 output tokens on Claude Sonnet 5.5:

• Input: 10,000 ÷ 1,000,000 × $2 = $0.02
• Output: 1,000 ÷ 1,000,000 × $10 = $0.01
• Total: $0.03

That looks tiny, until you make it 100,000 times a day. Because output is 5x the price per token, long answers are the expensive part. Asking for concise output is one of the cheapest optimisations there is. See LLM cost optimization for more.

#Trap 1: thinking tokens

Modern models can reason before they answer. That reasoning is made of tokens, and thinking is billed as output tokens. You pay for it whether or not the response shows it. On recent Claude models the thinking text is hidden by default, but the setting that controls visibility does not change what you are billed.

Practical consequences:

• A short visible answer can hide a large thinking bill.
• Thinking depth is controlled by an effort setting. Higher effort means more thinking and a bigger bill, and lower effort is often enough for chat and classification.
• Test effort levels on your own tasks instead of defaulting to the maximum.
• Set a sensible max_tokens so a runaway response cannot cost more than you expect.

#Trap 2: the model has no memory

A language model does not remember your last request. The API is stateless: every call must include the full conversation so far. The "memory" in a chat app is the app resending the history each time.

That means cost grows with conversation length. Suppose each turn adds about 500 tokens (your message plus the reply). Over 20 turns, the input you pay for is 500 × (0 + 1 + 2 + … + 19) = 95,000 tokens, even though only 10,000 tokens of new text were written. Add a long system prompt or documents and it multiplies.

Ways to control it:

• Trim or summarise old turns you no longer need.
• Keep the system prompt and tool list tight.
• Use prompt caching for the stable part (next section).
• For very long sessions, use the provider's compaction or context-editing features, and measure whether they actually save money for your workload.

#Prompt caching

If many requests start with the same large prefix (a system prompt, documents, tool definitions), caching lets the provider reuse that work. Cached input is read at a fraction of the normal input price.

For Claude, per Anthropic's documentation:

• Cache reads cost about 0.1x base input price (0.05x on Claude Opus 5.5, which is $0.20 per million tokens).
• Cache writes cost 1.25x for the default 5-minute lifetime, or 2x for the 1-hour option.
• A cache hit refreshes the timer, so requests arriving less than 5 minutes apart keep it warm.
• There is a minimum prefix length that varies by model, from 512 to 4,096 tokens. Shorter prefixes silently do not cache.
• It is a prefix match: change one byte early in the prompt and everything after it is recomputed. Put stable content first and volatile content (timestamps, user IDs, the new question) last.

#A worked example

A 50,000-token document prefix reused across 100 requests on Claude Sonnet 5.5, all within the cache lifetime:

• No caching: 5,000,000 input tokens × $2 per million = $10.00
• With caching: one write at 1.25x (about $0.13) plus 99 reads of 50,000 tokens at $0.20 per million (about $0.99) = about $1.12

Verify it works by checking cache_read_input_tokens in the response usage. If it is zero on repeated requests, something is changing the prefix.

#Batch processing

If nobody is waiting for the answer, use the batch API. Anthropic's Message Batches API processes requests asynchronously at 50% of standard prices, and the discount applies to every token, including cached ones.

• Up to 100,000 requests or 256 MB per batch.
• Most batches finish within an hour. The maximum is 24 hours.
• Results can arrive in any order, so match them by your own request ID.

Good fits are evaluation runs, backfills, classification of a large archive and nightly reports. A chat window is not a good fit.

#Picking the right model

Price differences between tiers are large: the same job can cost many times more on the top model than on a small one. Choose by measurement, not habit.

How to decide:

1. Build a test set of real cases. See our LLM evaluation guide.
2. Run the cheapest model first and score it.
3. Move up a tier only if the score is too low for the job.
4. Judge cost per completed task, not per request. A cheaper model that needs retries may cost more overall.
5. Try lower effort on a stronger model before adding a multi-model pipeline. One model also means one cache.

#A cost-cutting checklist

1. Count tokens before you ship, using the provider's endpoint.
2. Cap output with a sensible max_tokens and ask for concise answers.
3. Tune thinking effort per task.
4. Cache the stable prefix and keep it byte-identical.
5. Trim conversation history.
6. Batch anything that can wait.
7. Use the smallest model that passes your evaluation.
8. Log usage on every call and watch cost per task.

#Want a cheaper AI system?

We audit AI features for token waste, add caching and batching, and right-size models without hurting quality. Get a scope estimate for your AI agent, or see our AI agent development service and what AI agents cost to build.

#FAQ

#What is a token in an LLM?

A token is a chunk of text, often a short word or part of a word, that the model reads and writes. Providers bill per token, and every model has a limit on how many tokens it can handle at once.

#Why are output tokens more expensive than input tokens?

Generating text one token at a time is more costly than reading the prompt in one pass. On the Claude models listed here, output is priced at about 5x input.

#Are thinking tokens billed?

Yes. Thinking tokens are billed as output tokens even when the thinking text is hidden from the response.

#Why does the model forget what I said earlier?

It does not retain anything between calls. The API is stateless, so the application must resend the conversation history on every request, and you pay for those input tokens each time.

#How much does prompt caching save?

On Claude, cached reads cost about a tenth of normal input (less on some models), while writes cost 1.25x for the 5-minute option. With a large prefix reused many times, input cost can fall by around 90%.

#How much does the batch API save?

Anthropic's Message Batches API gives 50% off all token usage in exchange for asynchronous processing, with most batches finishing within an hour and a 24-hour maximum.

#How do I count tokens accurately?

Use the model provider's own counting endpoint with the model you will call. Estimates from another vendor's tokenizer can be off by 15 to 20% or more.

#Sources

• Anthropic: Pricing
• Anthropic: Prompt caching
• Anthropic: Batch processing
• Anthropic: Token counting