← Writing/llm-cache-is-the-real-price.mdx
·15 min read

The Cache Is the Real Price: What 4,500 API Calls Taught Us About LLM Costs

llm-costprompt-cachingkv-cacheagent-architectureexperiments

Everyone compares models on the price sheet: dollars per million input tokens, dollars per million output tokens.

For an AI agent, that leaves out something that can matter more than the model you pick: how much of each request the provider serves from its prompt cache. We wanted real numbers rather than theory, so we built one agent session and replayed it across three model families: about 4,500 API calls in all.


The experiment

  • One realistic agent session: a support/operations agent with a 6,000-word policy prompt and 16 tool definitions. Each turn adds a customer question and a block of tool output (order records as JSON).
  • 40 turns, growing from about 11,000–19,000 input tokens on turn 1 to 46,000–65,000 on turn 40, depending on the model.
  • Scripted, so every run sends the same workload. The models' replies are discarded and replaced by scripted ones, so only the model and the setting under test change.
  • Three model families, each with a large model and its smaller sibling:
FamilyLarge modelSmall model
Family AGPT-5.6 TerraGPT-5.6 Luna
Family BClaude Sonnet 5Claude Haiku 4.5
Family CGemini 2.5 ProGemini 2.5 Flash
  • Repeated: five runs of the baseline session per model, three of each routing policy, two of each reasoning and handoff policy and of each cached latency run. The uncached latency runs, each pause length and each shared-prompt user are single runs.
  • 4,505 API calls in 189 sessions, about $45. Every number below comes from the token counts each API reported, priced at each provider's list rates.

Each session opened with its own session ID, so conversations never shared a cache with each other. The system prompt and tools, identical for everyone, could still be shared across sessions, as they are in production.


Finding 1: caching makes an agent session 4.5–6.6× cheaper

Each 40-turn session, five times on every model:

ModelSession cost (mean, range over 5 runs)Same calls with no cacheSavingInput from cache, turns 2–40
A large$0.36 ($0.35–$0.36)$2.346.6×97%
A small$0.04 ($0.036–$0.044)$0.236.2×94–97%
B large$0.51$3.396.6×97%
B small$0.22$1.426.6×97%
C large$0.40 ($0.35–$0.47)$1.794.5×85–93%
C small$0.07 ($0.07–$0.08)$0.425.9×91–95%

"Same calls with no cache" is the token counts from these runs billed at the plain input price: exactly the calls we made, just without the discount.

Two things stand out:

  • The saving is large and consistent on families A and B: the five runs of each large model landed within 5% of each other. Family A's small model varied more, up to 23% between runs, with savings from 5.3× to 6.6×.
  • Family C's caching is best-effort. It caches automatically without being asked, and it usually hits, but not always. The same session varied by a third in cost between runs (85–93% of input from cache on the large model). You get most of the saving, but you can't plan on exactly how much.

Finding 2: the cache also makes long conversations faster

We re-ran the sessions with streaming, measuring time to first token, with reasoning switched off or set as low as the API allows:

ModelTurns 1–10: cached / uncachedTurns 11–20Turns 21–30Turns 31–40
A large0.83s / 0.87s0.83s / 0.89s0.91s / 0.97s0.95s / 1.06s
B large1.04s / 1.11s0.97s / 1.98s1.01s / 2.82s1.02s / 2.44s
B small0.73s / 0.76s0.73s / 0.88s0.69s / 0.98s0.69s / 1.12s
C small0.41s / 0.48s0.44s / 0.62s0.45s / 0.84s0.49s / 1.12s

(Medians, excluding each session's first turn. Family A's small model and family C's large model behaved like their siblings: a small gap on A, noisier on C, where the large model can't switch its thinking off entirely. On family C, our cache-busting runs still got a cache hit on a few calls, since its caching isn't something you can switch off.)

On families B and C, the cached response time stays flat as the conversation grows, while the uncached one climbs: up to 2.8× slower late in the session on family B's large model. Over the last ten turns, uncached was 2.4× slower on B's large model, 2.3× on C's small one and 1.6× on B's small one. Family A barely changes, which suggests its uncached reading is already fast at these sizes. Where caching helps latency, it helps most exactly when agents get slow: late in long sessions.


Finding 3: how long the cache lives varies a lot

We warmed a session on each large model for five turns, paused, then sent turn 6 (one run per pause length):

Pause before turn 61 min3 min4.5 min6 min10 min20 min35 min
Family A94%94%94%94%94%94%65%
Family B95%95%95%0%0%0%0%
Family C76%0%89%89%0%0%0%

Three very different behaviours:

  • Family A kept the whole cache through a 20-minute pause. At 35 minutes, the 65% it still had was exactly the shared system prompt and tools; the conversation itself was gone.
  • Family B has a hard cliff between 4.5 and 6 minutes, which matches its documented five-minute default. (It sells a one-hour cache at a higher write price.)
  • Family C was unpredictable: a 3-minute pause lost everything while a 6-minute pause kept 89%. That fits best-effort caching, where a hit depends on where your request happens to land, not only on time.

If your agent waits for a human, runs a slow tool, or works in the background, this table decides whether the next turn costs normal price or several times more.


Finding 4: "caching on" doesn't mean your system prompt is shared

Every user of an agent sends the same system prompt and tools: about 13,000 tokens of identical text. Ten different users, one first turn each:

SettingNew users (2–10): first-turn input from cache
Family A92%, automatically
Family B, automatic caching0%
Family B, one explicit cache marker after the system prompt (separate check, small model, users 2–3)91%
Family C0% on the large model; 0–80% (mean 27%) on the small one

Family B's automatic mode places its cache marker at the end of each request. That caches each conversation well (97%, see Finding 1), but it never saves an entry that ends where the shared system prompt ends, so a new user has nothing to reuse. In a separate check on family B's small model, adding one explicit marker after the system prompt took new users from 0% to 91%.

That's the kind of thing you'll only see if you measure: caching reported as on, and the most repeated tokens in your whole product being paid for in full on every new conversation.


Finding 5: "think harder" is mostly paid for in lost cache

Agents often raise the reasoning level for difficult steps. We tried two policies: each turn with a 25% chance of higher reasoning (it came out at about 30% of turns), and every fifth turn:

FamilySession cost vs never raising itShare of the extra cost that was input (cache misses and re-writes), not output
Family A+24% (every 5th) to +42% (random)96%
Family B+28% to +29%98–100%
Family C+69% to +74%34–56%

Changing the level cost cache on the changed turn. In our clean first-round test, the first switch got 0% from cache on both families we tried. In this larger round, where sessions ran back to back at the same levels, the turn on which the setting changed got 69–89% of its input from cache, against 81–97% on unchanged turns (token-weighted, by family and policy): part of the prompt was often still cached at that level from earlier turns or sessions. One provider's documentation explains the mechanism: the reasoning setting is rendered into the prompt, so a different setting means a different prompt.

The surprise is how much of the extra cost that is. On families A and B, almost all of it was input cost: cache misses and re-writing the cache, not the output. Family C's output grew much more, so input made up a smaller share. We asked for one-sentence replies, so on real tasks output (including reasoning) will be a larger share everywhere, but the input penalty doesn't go away.


Finding 6: switching models costs more than planned, unless you switch rarely

The classic cost trick: send "easy" turns to the small model. We tried four policies over 40 turns, three runs each. The random policies gave each turn a 25%, 50% or 75% chance of going to the small model (it came out at 28%, 40% and 82% of turns):

PolicyModel switches per session (mean)Family A: hidden cache cost / saving vs all largeFamily BFamily C
25% chance easy (random)16+11% / 17%+23% / −1%+12% / 13%
50% chance easy (random)19+13% / 28%+23% / 6%+3% / 29%
75% chance easy (random)11+60% / 56%+47% / 23%+34% / 57%
Switch at most every 10 turns1+3% / 38%+15% / 16%+5% / 38%

"Hidden cache cost" compares the real bill with what you'd plan from each model's normal per-turn cost in a warm session. Every policy cost more than that plan, because each switch meant the other model had to catch up.

  • Per-turn random routing on family B cost more than never switching at the 25% setting, and saved 6% at 50%.
  • Switching at most once every ten turns had the smallest hidden cost on families A and B. On family C, random 50% (+3%) and the ten-turn policy (+5%) were both small.
  • The savings depend heavily on the price gap: family A's small model costs a tenth of its large one, family B's about half.

(These are cost numbers only. Whether the small model handles those turns well enough is a separate question.)


Finding 7: if you must switch, hand over a summary

We moved the conversation from the large to the small model at turn 21, once with the full transcript, once with a short written summary of turns 1–20:

Cost of turns 21–40Full transcriptSummary insteadSaving
Family A$0.027$0.01642%
Family B$0.161$0.09442%
Family C$0.059$0.02952%

For turns 21–40, the summary was even 22–33% cheaper than having used the small model from the start, simply because the conversation it carries is shorter. (Over the whole session, starting on the small model is still cheaper, because the first 20 turns ran on the large one.) The catches: a summary drops detail, we didn't measure what that costs in answer quality, and our summary was scripted, so the cost of generating it isn't included.


From our first, smaller round

Our first round (20 turns, families A and B) found a few more ways to lose the cache:

  • A timestamp at the top of the system prompt: 0% from cache for the entire session, 6.6–7.1× the normal cost, with nothing visibly wrong.
  • Reordering the tool list mid-session: 0% on that turn.
  • Token counts: the same prompt came to about 11,300 tokens on one model and 18,800 on another, and even two models from the same provider differed by 29%. The counts include each API's formatting, so this isn't purely the tokeniser, but either way per-token prices don't compare cleanly across models.

What we'd do with this

Steal this:

  1. Measure your cache hit rate. All three families report cached tokens in every response. Log them next to cost and latency. It's the only way you'll notice most of the problems above.
  2. Make sure the shared part is cached for new users. If your provider uses explicit cache markers, put one right after the system prompt and tools, not only at the end.
  3. Keep the front of the prompt stable. Tools and system instructions first, the conversation after, anything that changes last. No timestamps up front.
  4. Treat reasoning level and model as session decisions. Change them rarely. If one step needs heavy reasoning or a different model, give it its own short context.
  5. Know your cache lifetime. Minutes on some providers, much longer on others, and unpredictable on best-effort caches. Pauses longer than that cost several times a normal turn.
  6. If you switch models, switch rarely and hand over a summary. Rare switches kept the hidden cost down; a summary cut the cost of the remaining turns by 42–52%.
  7. Price the whole session, not the token. The same workload's cost moved by up to 7× depending on caching alone.

Next: smart router or smart harness?

The obvious response to expensive models is routing: send easy work to cheap models and hard work to expensive ones. Several teams are building routers that try to account for exactly these effects.

The other approach is to change the harness around the model: how the prompt is laid out, where the cache markers go, how context is split between sub-agents, when to summarise, and what stays stable.

This data points at both. Which one actually cuts the bill? That's the next post.


Claim ledger

Every number above comes from these files, published alongside this post, or from the dated provider pages listed below. The workload is synthetic; the files contain token counts, costs and timings, no conversation content from real users.

  • summary-v2.json: the 4,505-call round, one section per finding.
  • derived.json: figures computed from the raw per-call records that the summary doesn't hold directly (reasoning cost split, changed-turn cache shares, token counts).
  • explicit-breakpoint-check.json: the three-user explicit cache-marker check.
  • summary-round-1.json: the first, 20-turn round on families A and B.
  • manifest.json: the workload (policy length, tool count, turns) and run totals (calls, sessions, failures, spend).
ClaimEvidence
Caching makes the 40-turn session 4.5–6.6× cheaper; 85–97% of input from cachesummary-v2.json, scale
Cached time to first token stays flat; uncached up to 2.8× slowersummary-v2.json, latency
Cache lifetime by pause lengthsummary-v2.json, ttl
New users: 92% (A), 0% (B, automatic), 0% or unpredictable (C)summary-v2.json, shared_prefix
One explicit marker after the system prompt: 91% for new usersexplicit-breakpoint-check.json
Raising reasoning: +24% to +74% session costsummary-v2.json, reasoning_policy
Share of that extra cost that was input; cache shares on changed vs unchanged turnsderived.json, reasoning_policy
Routing policies: hidden cache cost and saving vs all largesummary-v2.json, router
Summary handoff: turns 21–40 cost 42–52% lesssummary-v2.json, handoff
Timestamp and tool-reorder resultssummary-round-1.json, breakers; cost ratios in derived.json, round_1_timestamp_in_system_prompt
Same prompt, different token counts (including the 29% gap)derived.json, token_counts_turn1
Workload size, call and session counts, spendmanifest.json
List prices, including cache reads and writes[1][2][3], checked 27–28 September 2026
The reasoning setting is rendered into the prompt; five-minute default cache lifetime and a paid one-hour option (family B)[4]
Family A's cache write pricing and reporting[5]
Family C caches automatically, best-effort[6]

[1] OpenAI, API pricing

[2] Anthropic, Pricing

[3] Google, Gemini API pricing

[4] Anthropic, Prompt caching

[5] OpenAI, Prompt caching

[6] Google, Context caching


Method: one scripted 40-turn session; 4,505 API calls in 189 sessions (one failed call, and its session was excluded from the results). Costs come from the token counts each API reported, at list prices checked on 27–29 September 2026. Cached input is billed at a tenth of the input price on all three families; cache writes at 1.25× input on families A and B; family C charges no write fee for its automatic cache. (We first used a misread write price for family A's small model; its calls were re-priced from their recorded token counts.) We requested one-sentence replies, so output costs are small compared with real tasks. Quality was not measured. Latency is from one location and one period of time. The earlier 20-turn round used 492 calls on families A and B. Your prompts, providers and traffic will differ.