LLM caching, batching and routing

LLM caching, batching and routing each cut a different term of a request's cost: prompt caching makes the repeated start of the input cheaper, batching halves the rate at OpenAI, Anthropic and Google for work that needs no immediate answer, and routing between models picks a cheaper model per request. A cache in the application avoids the request altogether. Which lever applies depends on the workload: a stable prefix, a wait the user accepts, or requests a smaller model answers just as well.

How it works

The cached and batch rates themselves are in tokens, requests and context; this page covers when they apply.

Prompt caching at the providers. All three cache only an exactly matching prefix, the start of the request up to a breakpoint. To the model, Google explains, the cached part is simply the start of the prompt: caching changes the price, not the answer.

OpenAI and Anthropic extend the lifetime on every hit without a new write charge. OpenAI keeps caches within one organisation and processing region, Anthropic on the Claude API within one workspace.

Caching in the application. A cache in front of the model answers a repeated request from a stored response; an exact cache matches the prompt character by character, a semantic cache its meaning through embeddings. Azure API Management documents the semantic variant: each lookup calls an embeddings deployment, responses sit in Azure Managed Redis, a score threshold sets how similar a prompt must be, and vary-by partitions the cache, for example per subscription. That costs an embedding per request and the cache store (embedding costs, retrieval and vector databases) and saves the model call on every hit.

Batch. OpenAI, Anthropic and Google bill asynchronous batch jobs at half price. OpenAI allows only a 24-hour window, with separate, higher rate limits; at expiry, unfinished requests are cancelled and completed ones billed. Anthropic completes most batches within an hour, expires them after 24 hours, and does not bill errored, cancelled or expired requests. Cache multipliers stack with the batch discount there, but cache hits in a batch are best effort, so Anthropic suggests the one-hour cache for shared context. Google gives 24 hours as its target and expires a job after 48 hours without results. Through the clouds, Amazon Bedrock offers batch at 50 percent below on-demand for selected models and Azure OpenAI at 50 percent below Global Standard within 24 hours.

Routing. Routing can follow the application's own rules, per feature or input length, or a provider's router. Intelligent prompt routing in Amazon Bedrock predicts the response quality of the models in one family for each request and chooses one, with a fallback model and a threshold for the quality difference; the response names the model used. AWS notes it is optimised for English prompts only and on 2026-10-16 priced it at USD 1 per 1,000 requests, with no region given. The model router in Microsoft Foundry is deployed like a model, with Balanced, Cost and Quality modes and a selectable subset of models; the response's model field names the model that served it. Its cost is that of the models used plus a router charge on input tokens, which Microsoft's how-to page leaves out; on 2026-10-16 the Azure Retail Prices API listed it for Sweden Central at USD 0.14 per million tokens for Global and USD 0.154 for Data Zone, effective since 2026-01-01.

What it means for your cost

The order follows how far each lever reaches: caching leaves the answer unchanged and comes first, batch moves only the timing, routing changes the model and needs a quality check. Usage data shows each lever separately: cache read and write counters, a batch rate, and the model that served a routed request (AI provider usage exports). Rates per model are in AI model pricing and rate cards; for agents, caching acts on the definitions and history each round resends (LLM tool call pricing and agent loops).

Where it goes wrong

• Variable data at the start of the prompt. Every request writes to the cache and none reads. The cause: timestamps or user data sit before the stable part. Anthropic advises a breakpoint at the end of the static prefix, since automatic caching falls into the same trap.
• Cache entries nobody reads. Usage data shows write tokens with hardly any reads. The cause: gaps longer than the lifetime, parallel requests sent before the first response (at Anthropic an entry is usable only once the first response begins), or prompts below the minimum, which Anthropic processes uncached without an error.
• Batch for work that needs an answer now. Users wait, or the work drifts back to synchronous requests at full price. The cause: batch chosen for the discount instead of the deadline.
• Routing without a quality check. Cost per request falls while follow-up requests rise. The cause: nobody compares the answers. Microsoft advises benchmarking the router against the current model first; automatic updates swap the underlying models and their cost, and caching stops helping when consecutive requests land on different models.
• A semantic cache without partitions. One customer's answer reaches another. The cause: no vary-by per user or tenant.

Example

When a cache entry pays off, at rates read on 2026-10-16, Anthropic on the Claude API. n counts reads; the first two rows are multiples of the input rate, the last USD per million tokens, excluding the request that creates the cache.

With the 0.05 read rate of gpt-6.1-sol, Claude Opus 5.5 and Sonnet 5.5 the thresholds stay the same.

Sources

Read on 2026-10-16. OpenAI prompt caching, Batch API and pricing; Anthropic prompt caching, batch processing and pricing; Google context caching, Batch API and Gemini API pricing; AWS intelligent prompt routing and Amazon Bedrock pricing; Microsoft model router, use model router, Azure OpenAI pricing, the Azure Retail Prices API for Sweden Central, semantic caching in Azure API Management and the semantic cache lookup policy.