The Cache Is the Real Price: What 4,500 API Calls Taught Us About LLM Costs
Everyone compares models on the price sheet: dollars per million input tokens, dollars per million output tokens.
For an AI agent, that leaves out something that can matter more than the model you pick: how much of each request the provider serves from its prompt cache. We wanted real numbers rather than theory, so we built one agent session and replayed it across three model families: about 4,500 API calls in all.
The experiment
- One realistic agent session: a support/operations agent with a 6,000-word policy prompt and 16 tool definitions. Each turn adds a customer question and a block of tool output (order records as JSON).
- 40 turns, growing from about 11,000–19,000 input tokens on turn 1 to 46,000–65,000 on turn 40, depending on the model.
- Scripted, so every run sends the same workload. The models' replies are discarded and replaced by scripted ones, so only the model and the setting under test change.
- Three model families, each with a large model and its smaller sibling:
| Family | Large model | Small model |
|---|---|---|
| Family A | GPT-5.6 Terra | GPT-5.6 Luna |
| Family B | Claude Sonnet 5 | Claude Haiku 4.5 |
| Family C | Gemini 2.5 Pro | Gemini 2.5 Flash |
- Repeated: five runs of the baseline session per model, three of each routing policy, two of each reasoning and handoff policy and of each cached latency run. The uncached latency runs, each pause length and each shared-prompt user are single runs.
- 4,505 API calls in 189 sessions, about $45. Every number below comes from the token counts each API reported, priced at each provider's list rates.
Each session opened with its own session ID, so conversations never shared a cache with each other. The system prompt and tools, identical for everyone, could still be shared across sessions, as they are in production.
Finding 1: caching makes an agent session 4.5–6.6× cheaper
Each 40-turn session, five times on every model:
| Model | Session cost (mean, range over 5 runs) | Same calls with no cache | Saving | Input from cache, turns 2–40 |
|---|---|---|---|---|
| A large | $0.36 ($0.35–$0.36) | $2.34 | 6.6× | 97% |
| A small | $0.04 ($0.036–$0.044) | $0.23 | 6.2× | 94–97% |
| B large | $0.51 | $3.39 | 6.6× | 97% |
| B small | $0.22 | $1.42 | 6.6× | 97% |
| C large | $0.40 ($0.35–$0.47) | $1.79 | 4.5× | 85–93% |
| C small | $0.07 ($0.07–$0.08) | $0.42 | 5.9× | 91–95% |
"Same calls with no cache" is the token counts from these runs billed at the plain input price: exactly the calls we made, just without the discount.
Two things stand out:
- The saving is large and consistent on families A and B: the five runs of each large model landed within 5% of each other. Family A's small model varied more, up to 23% between runs, with savings from 5.3× to 6.6×.
- Family C's caching is best-effort. It caches automatically without being asked, and it usually hits, but not always. The same session varied by a third in cost between runs (85–93% of input from cache on the large model). You get most of the saving, but you can't plan on exactly how much.
Finding 2: the cache also makes long conversations faster
We re-ran the sessions with streaming, measuring time to first token, with reasoning switched off or set as low as the API allows:
| Model | Turns 1–10: cached / uncached | Turns 11–20 | Turns 21–30 | Turns 31–40 |
|---|---|---|---|---|
| A large | 0.83s / 0.87s | 0.83s / 0.89s | 0.91s / 0.97s | 0.95s / 1.06s |
| B large | 1.04s / 1.11s | 0.97s / 1.98s | 1.01s / 2.82s | 1.02s / 2.44s |
| B small | 0.73s / 0.76s | 0.73s / 0.88s | 0.69s / 0.98s | 0.69s / 1.12s |
| C small | 0.41s / 0.48s | 0.44s / 0.62s | 0.45s / 0.84s | 0.49s / 1.12s |
(Medians, excluding each session's first turn. Family A's small model and family C's large model behaved like their siblings: a small gap on A, noisier on C, where the large model can't switch its thinking off entirely. On family C, our cache-busting runs still got a cache hit on a few calls, since its caching isn't something you can switch off.)
On families B and C, the cached response time stays flat as the conversation grows, while the uncached one climbs: up to 2.8× slower late in the session on family B's large model. Over the last ten turns, uncached was 2.4× slower on B's large model, 2.3× on C's small one and 1.6× on B's small one. Family A barely changes, which suggests its uncached reading is already fast at these sizes. Where caching helps latency, it helps most exactly when agents get slow: late in long sessions.
Finding 3: how long the cache lives varies a lot
We warmed a session on each large model for five turns, paused, then sent turn 6 (one run per pause length):
| Pause before turn 6 | 1 min | 3 min | 4.5 min | 6 min | 10 min | 20 min | 35 min |
|---|---|---|---|---|---|---|---|
| Family A | 94% | 94% | 94% | 94% | 94% | 94% | 65% |
| Family B | 95% | 95% | 95% | 0% | 0% | 0% | 0% |
| Family C | 76% | 0% | 89% | 89% | 0% | 0% | 0% |
Three very different behaviours:
- Family A kept the whole cache through a 20-minute pause. At 35 minutes, the 65% it still had was exactly the shared system prompt and tools; the conversation itself was gone.
- Family B has a hard cliff between 4.5 and 6 minutes, which matches its documented five-minute default. (It sells a one-hour cache at a higher write price.)
- Family C was unpredictable: a 3-minute pause lost everything while a 6-minute pause kept 89%. That fits best-effort caching, where a hit depends on where your request happens to land, not only on time.
If your agent waits for a human, runs a slow tool, or works in the background, this table decides whether the next turn costs normal price or several times more.
Finding 4: "caching on" doesn't mean your system prompt is shared
Every user of an agent sends the same system prompt and tools: about 13,000 tokens of identical text. Ten different users, one first turn each:
| Setting | New users (2–10): first-turn input from cache |
|---|---|
| Family A | 92%, automatically |
| Family B, automatic caching | 0% |
| Family B, one explicit cache marker after the system prompt (separate check, small model, users 2–3) | 91% |
| Family C | 0% on the large model; 0–80% (mean 27%) on the small one |
Family B's automatic mode places its cache marker at the end of each request. That caches each conversation well (97%, see Finding 1), but it never saves an entry that ends where the shared system prompt ends, so a new user has nothing to reuse. In a separate check on family B's small model, adding one explicit marker after the system prompt took new users from 0% to 91%.
That's the kind of thing you'll only see if you measure: caching reported as on, and the most repeated tokens in your whole product being paid for in full on every new conversation.
Finding 5: "think harder" is mostly paid for in lost cache
Agents often raise the reasoning level for difficult steps. We tried two policies: each turn with a 25% chance of higher reasoning (it came out at about 30% of turns), and every fifth turn:
| Family | Session cost vs never raising it | Share of the extra cost that was input (cache misses and re-writes), not output |
|---|---|---|
| Family A | +24% (every 5th) to +42% (random) | 96% |
| Family B | +28% to +29% | 98–100% |
| Family C | +69% to +74% | 34–56% |
Changing the level cost cache on the changed turn. In our clean first-round test, the first switch got 0% from cache on both families we tried. In this larger round, where sessions ran back to back at the same levels, the turn on which the setting changed got 69–89% of its input from cache, against 81–97% on unchanged turns (token-weighted, by family and policy): part of the prompt was often still cached at that level from earlier turns or sessions. One provider's documentation explains the mechanism: the reasoning setting is rendered into the prompt, so a different setting means a different prompt.
The surprise is how much of the extra cost that is. On families A and B, almost all of it was input cost: cache misses and re-writing the cache, not the output. Family C's output grew much more, so input made up a smaller share. We asked for one-sentence replies, so on real tasks output (including reasoning) will be a larger share everywhere, but the input penalty doesn't go away.
Finding 6: switching models costs more than planned, unless you switch rarely
The classic cost trick: send "easy" turns to the small model. We tried four policies over 40 turns, three runs each. The random policies gave each turn a 25%, 50% or 75% chance of going to the small model (it came out at 28%, 40% and 82% of turns):
| Policy | Model switches per session (mean) | Family A: hidden cache cost / saving vs all large | Family B | Family C |
|---|---|---|---|---|
| 25% chance easy (random) | 16 | +11% / 17% | +23% / −1% | +12% / 13% |
| 50% chance easy (random) | 19 | +13% / 28% | +23% / 6% | +3% / 29% |
| 75% chance easy (random) | 11 | +60% / 56% | +47% / 23% | +34% / 57% |
| Switch at most every 10 turns | 1 | +3% / 38% | +15% / 16% | +5% / 38% |
"Hidden cache cost" compares the real bill with what you'd plan from each model's normal per-turn cost in a warm session. Every policy cost more than that plan, because each switch meant the other model had to catch up.
- Per-turn random routing on family B cost more than never switching at the 25% setting, and saved 6% at 50%.
- Switching at most once every ten turns had the smallest hidden cost on families A and B. On family C, random 50% (+3%) and the ten-turn policy (+5%) were both small.
- The savings depend heavily on the price gap: family A's small model costs a tenth of its large one, family B's about half.
(These are cost numbers only. Whether the small model handles those turns well enough is a separate question.)
Finding 7: if you must switch, hand over a summary
We moved the conversation from the large to the small model at turn 21, once with the full transcript, once with a short written summary of turns 1–20:
| Cost of turns 21–40 | Full transcript | Summary instead | Saving |
|---|---|---|---|
| Family A | $0.027 | $0.016 | 42% |
| Family B | $0.161 | $0.094 | 42% |
| Family C | $0.059 | $0.029 | 52% |
For turns 21–40, the summary was even 22–33% cheaper than having used the small model from the start, simply because the conversation it carries is shorter. (Over the whole session, starting on the small model is still cheaper, because the first 20 turns ran on the large one.) The catches: a summary drops detail, we didn't measure what that costs in answer quality, and our summary was scripted, so the cost of generating it isn't included.
From our first, smaller round
Our first round (20 turns, families A and B) found a few more ways to lose the cache:
- A timestamp at the top of the system prompt: 0% from cache for the entire session, 6.6–7.1× the normal cost, with nothing visibly wrong.
- Reordering the tool list mid-session: 0% on that turn.
- Token counts: the same prompt came to about 11,300 tokens on one model and 18,800 on another, and even two models from the same provider differed by 29%. The counts include each API's formatting, so this isn't purely the tokeniser, but either way per-token prices don't compare cleanly across models.
What we'd do with this
Steal this:
- Measure your cache hit rate. All three families report cached tokens in every response. Log them next to cost and latency. It's the only way you'll notice most of the problems above.
- Make sure the shared part is cached for new users. If your provider uses explicit cache markers, put one right after the system prompt and tools, not only at the end.
- Keep the front of the prompt stable. Tools and system instructions first, the conversation after, anything that changes last. No timestamps up front.
- Treat reasoning level and model as session decisions. Change them rarely. If one step needs heavy reasoning or a different model, give it its own short context.
- Know your cache lifetime. Minutes on some providers, much longer on others, and unpredictable on best-effort caches. Pauses longer than that cost several times a normal turn.
- If you switch models, switch rarely and hand over a summary. Rare switches kept the hidden cost down; a summary cut the cost of the remaining turns by 42–52%.
- Price the whole session, not the token. The same workload's cost moved by up to 7× depending on caching alone.
Next: smart router or smart harness?
The obvious response to expensive models is routing: send easy work to cheap models and hard work to expensive ones. Several teams are building routers that try to account for exactly these effects.
The other approach is to change the harness around the model: how the prompt is laid out, where the cache markers go, how context is split between sub-agents, when to summarise, and what stays stable.
This data points at both. Which one actually cuts the bill? That's the next post.
Claim ledger
Every number above comes from these files, published alongside this post, or from the dated provider pages listed below. The workload is synthetic; the files contain token counts, costs and timings, no conversation content from real users.
- summary-v2.json: the 4,505-call round, one section per finding.
- derived.json: figures computed from the raw per-call records that the summary doesn't hold directly (reasoning cost split, changed-turn cache shares, token counts).
- explicit-breakpoint-check.json: the three-user explicit cache-marker check.
- summary-round-1.json: the first, 20-turn round on families A and B.
- manifest.json: the workload (policy length, tool count, turns) and run totals (calls, sessions, failures, spend).
| Claim | Evidence |
|---|---|
| Caching makes the 40-turn session 4.5–6.6× cheaper; 85–97% of input from cache | summary-v2.json, scale |
| Cached time to first token stays flat; uncached up to 2.8× slower | summary-v2.json, latency |
| Cache lifetime by pause length | summary-v2.json, ttl |
| New users: 92% (A), 0% (B, automatic), 0% or unpredictable (C) | summary-v2.json, shared_prefix |
| One explicit marker after the system prompt: 91% for new users | explicit-breakpoint-check.json |
| Raising reasoning: +24% to +74% session cost | summary-v2.json, reasoning_policy |
| Share of that extra cost that was input; cache shares on changed vs unchanged turns | derived.json, reasoning_policy |
| Routing policies: hidden cache cost and saving vs all large | summary-v2.json, router |
| Summary handoff: turns 21–40 cost 42–52% less | summary-v2.json, handoff |
| Timestamp and tool-reorder results | summary-round-1.json, breakers; cost ratios in derived.json, round_1_timestamp_in_system_prompt |
| Same prompt, different token counts (including the 29% gap) | derived.json, token_counts_turn1 |
| Workload size, call and session counts, spend | manifest.json |
| List prices, including cache reads and writes | [1][2][3], checked 27–28 September 2026 |
| The reasoning setting is rendered into the prompt; five-minute default cache lifetime and a paid one-hour option (family B) | [4] |
| Family A's cache write pricing and reporting | [5] |
| Family C caches automatically, best-effort | [6] |
[1] OpenAI, API pricing
[2] Anthropic, Pricing
[3] Google, Gemini API pricing
[4] Anthropic, Prompt caching
[5] OpenAI, Prompt caching
[6] Google, Context caching
Method: one scripted 40-turn session; 4,505 API calls in 189 sessions (one failed call, and its session was excluded from the results). Costs come from the token counts each API reported, at list prices checked on 27–29 September 2026. Cached input is billed at a tenth of the input price on all three families; cache writes at 1.25× input on families A and B; family C charges no write fee for its automatic cache. (We first used a misread write price for family A's small model; its calls were re-priced from their recorded token counts.) We requested one-sentence replies, so output costs are small compared with real tasks. Quality was not measured. Latency is from one location and one period of time. The earlier 20-turn round used 492 calls on families A and B. Your prompts, providers and traffic will differ.