The cache you think you have

Caching bugs are quiet. Nothing errors. Nothing crashes. The system just costs four times what it should and everyone assumes that's the price of the feature.

The reason is structural: a cache miss is indistinguishable from a cache that was never there. There is no exception, no log line, no failing test. The only symptom is a number in a dashboard nobody built. So the first move — before any tuning — is always the same: measure the hit rate, or you are guessing.

Here is a tour of the ways caches silently don't work, in roughly ascending order of how long they take to find.

Level 1: the key varies and you didn't notice#

Every cache is a hash map, and the entire game is whether two requests that should share an entry produce the same key.

The classic HTTP version is Vary. Your CDN caches a response, and someone adds Vary: Accept-Encoding, User-Agent to the headers. Now the cache is keyed by user agent, and since there are effectively infinite user agent strings, your hit rate goes to approximately zero. The response is still correct. It is just never cached.

The application version is worse because it hides in serialization:

# These produce different cache keys for identical data.
key = f"user:{user_id}:prefs:{json.dumps(prefs)}"

json.dumps on a dict preserves insertion order. Build prefs from a database row one time and from a request body another time, and the same preferences serialize to different strings. sort_keys=True fixes it. Nothing tells you it was broken.

Same family of bug: iterating a set to build a key, including a float that formats differently across platforms, or embedding a timestamp with second precision into something meant to be cached for an hour.

Level 2: the prefix problem#

Prefix caches — the kind used for LLM prompts — have a stricter rule than hash caches, and it catches people who have cached things successfully for years.

The cache key is derived from the exact bytes up to each breakpoint. Any change anywhere in the prefix invalidates everything after it.

Not the changed part. Everything after it. So this innocuous line:

system = f"You are a helpful assistant. Current date: {datetime.now()}."

means you have never had a single cache hit, on any request, ever. The date changes every call, it sits at the very front of the prefix, and everything downstream — your tools, your 30-turn conversation history, all of it — gets re-processed at full price on every turn.

The rendering order for a Claude request is tools → system → messages, and that ordering is the whole design constraint. Anything volatile has to go after everything stable, which in practice means:

• No timestamps, request IDs, or user IDs in the system prompt.
• Tools serialized deterministically, and the tool list not varying per user.
• Model held constant — caches are per-model, so switching models mid-session starts from cold.
• Conditional system prompt sections avoided, since every flag combination is a distinct prefix.

The diagnostic is direct. The response tells you what happened:

u = response.usage
print(u.cache_creation_input_tokens)  # written this request (~1.25x cost)
print(u.cache_read_input_tokens)      # served from cache (~0.1x cost)
print(u.input_tokens)                 # full price — the uncached remainder

If cache_read_input_tokens is zero across repeated requests with what you believe is an identical prefix, something in that prefix is not identical. Diff the rendered bytes of two consecutive requests. The culprit is always in the first few hundred characters, and it is almost always a timestamp.

Level 3: the thing you're caching isn't the expensive thing#

A team I worked with cached the result of an expensive aggregation query with a five-minute TTL. Hit rate: 96%. Beautiful. Latency: unchanged.

The aggregation was 30ms. The 400ms was three N+1 queries in the serializer that ran on both the hit and the miss path. They had cached the cheap half of the request perfectly.

Before caching anything, profile the thing you are about to cache and confirm it is actually where the time goes. Then profile again after, because the remaining work is now a larger fraction of the total and the next bottleneck is usually somewhere you didn't expect.

Level 4: the stampede#

Everything works until the cache entry expires under load.

One popular key expires. Two hundred concurrent requests all miss simultaneously. All two hundred run the expensive query. The database, which was comfortable serving one of these per five minutes, now gets two hundred at once and falls over. The cache repopulates, the site recovers, and it happens again in five minutes.

Three fixes, in increasing order of robustness:

1. Jitter the TTL. 300 + random.randint(0, 60) seconds. Costs one line, solves the synchronized-expiry case where many keys were populated at once.
2. Single-flight. First miss takes a lock and computes; the rest wait for that result rather than duplicating the work. Most cache libraries have this; it is often not on by default.
3. Serve stale while revalidating. On a miss, return the expired value immediately and refresh in the background. Latency stays flat and the origin sees exactly one request. This is what stale-while-revalidate does in HTTP, and it is the right default for anything where slightly stale is acceptable.

There's an LLM-shaped version of this too. Fire N parallel requests sharing a large prefix and all N pay full price — a cache entry only becomes readable once the first response starts streaming. Send one, wait for the first token, then fan out the rest.

Level 5: your database's cache, which you forgot exists#

Postgres has a buffer cache, and its hit ratio is the single most useful number nobody looks at:

SELECT
  sum(heap_blks_hit) / nullif(sum(heap_blks_hit + heap_blks_read), 0) AS ratio
FROM pg_statio_user_tables;

Below ~0.99 on an OLTP workload means you are going to disk far more than you should. Usually the fix is not more RAM — it is a missing index causing sequential scans that evict everything useful from the cache. One bad query can degrade the performance of every other query on the box by trashing shared buffers, which is why "the site got slow but this one endpoint got slower" is such a common shape.

The pattern underneath#

Every one of these is the same mistake at a different layer: assuming the cache key is what you think it is, and never checking.

Caches don't tell you when they're not working. They just quietly do nothing while you pay for the machinery. So instrument the hit rate first, before tuning anything. Then when someone asks whether the cache is helping, you can answer with a number instead of a shrug — which is, in the end, the only thing separating performance work from superstition.