Prompt caching isn't the optimization you think it is

Everyone says the same thing when your LLM bill gets scary: “enable prompt caching.” We did. It saved us about 2%.

Then we moved a cheap classification step in front of the big model, and costs dropped by half. This is the story of that second thing — and why it matters more than the first.

What’s a decision model?

A decision model (Jev, Clef, etc.) is a small, cheap model (or sometimes just a rules function) that makes one narrow decision before you call the big, expensive model.

It doesn’t answer the user. It decides how the thing that answers should be configured:

• which context to include,

• which tools to load,

• how much “thinking” to spend.

It’s a router, not a brain. And because it’s tiny, it costs almost nothing and adds only tens of milliseconds. That’s the whole idea — make a cheap decision so the expensive call doesn’t have to.

The problem

Our assistant handled a dozen different jobs: refunds, billing, account recovery, bug triage, how-to guides, deep product analytics. Each job had its own hefty block of system-prompt instructions — rules, examples, edge cases.

The naive design shoved every block into every request. Two costs followed:

1. Context cost. Every single call — even “how do I reset my password?” — paid for all dozen instruction blocks.

2. Reasoning cost. The model was allowed to “think” (hidden chain-of-thought) on every request, whether it was a one-word lookup or a genuine analysis. Deep reasoning is where output tokens explode.

The result: a simple question and a hard question cost roughly the same amount, because they both dragged the full prompt and full reasoning along.

Why caching wasn’t enough

Prompt caching discounts repeated input tokens — the static prefix — by ~90%. That sounds enormous until you look at where the tokens actually are.

Caching has two blind spots:

1. It only touches the static prefix. The feature blocks change with each request (a refund request needs the refund rules, not the analytics rules), so they can’t be cached. They’re dynamic input.

2. It doesn’t touch output tokens at all. And output — hidden reasoning plus the visible answer — is where the real money lives.

Here’s one “deep” request, broken down by tokens:

We were optimizing the 7,000-token slice while ignoring the 21,000 tokens next to it. Caching was necessary, but it was fixing 5% of the problem.

The solution: route it

So we put a decision model in front.

On every request, a tiny classifier looks at the raw user message and outputs a handful of flags:

• Which modules are relevant → inject only those.

• How much reasoning to spend → deep or light.

The main model then gets a lean, tailored prompt: the static core plus only the relevant modules, with reasoning switched off for light requests.

The classifier runs in ~100–300ms and costs a fraction of a cent. It turned a monolith into a router.

Why this helps even when you have caching

The two techniques don’t compete — they compound, because they target different things:

• Caching discounts the static core (the part that repeats).

• The decision model removes the parts caching can’t reach: the dynamic modules and the reasoning tokens.

Light requests collapsed by ~70% — the whole reasoning budget vanished, and only one small module rode along. Deep requests shrank a more modest ~10%, because the hard work is genuinely hard and still needs the reasoning; we only trimmed the irrelevant context.

And caching still does its job on the smaller static core, so the two savings stack.

The takeaway

Before you reach for caching, ask: what am I actually spending tokens on? If the answer is “output/reasoning” or “per-request context,” caching won’t save you — you need to not spend those tokens in the first place.

A decision model is the cheapest way to do that. Put a small, fast classifier in front of the big model. Let it decide what context matters and how hard to think. Then pay the big model only for the work the request actually needs.

Caching optimizes the price of what you ship. A decision model lets you ship less.