Caching Strategies

The 1 ms vs 200 ms problem

Sarah opens Instagram. Her feed appears in under a second — 30 posts, each with a photo, a username, a comment count. Now imagine each of those numbers required a fresh database query. Thirty round-trips, each maybe 5–20 ms inside the data center. At scale, with millions of Sarahs all scrolling at the same second, the database simply melts. Caching is the answer to that meltdown — but it is also the source of about half the production outages in your career. So we'd better get it right.

The numbers that drive every cache decision

Before any pattern makes sense, you have to feel these numbers in your bones. They're the reason caching is not optional at scale.

Read that table twice. A Redis hit is roughly 20× faster than a database query. A process-local cache hit is 5000× faster. Caching is not a "nice to have" optimization — at scale it is the only thing keeping the response time inside the second budget your users will tolerate before they leave.

The two laws of caching

Where caches actually live

When Sarah's browser shows her feed, the bytes she's seeing may have passed through up to five caches in series before reaching her eyes. Each layer is a chance to short-circuit a request and never touch the database. Knowing where each layer sits — and what it's good at — is half the battle.

The six caching patterns every system uses

"Caching strategy" is shorthand for two related decisions: how does the cache get filled? and what happens on a write? There are six classic patterns, and almost every cache in production is one of them — or a hybrid of two. We'll walk through each with a sequence diagram, the code you'd actually write, and the trap that catches engineers the first time they try it.

Pattern 1 — Cache-Aside (a.k.a. Lazy Loading)

The most common pattern in the wild. The application code talks to both the cache and the database — the cache is "off to the side", and the app is the conductor.

public User getUser(long id) {
  String key = "user:" + id;
  User cached = redis.get(key, User.class);
  if (cached != null) return cached;        // HIT

  User fromDb = userRepo.findById(id);          // MISS → DB
  if (fromDb != null) {
    redis.setex(key, 300, fromDb);            // fill cache with 5-min TTL
  }
  return fromDb;
}

public void updateUser(User u) {
  userRepo.save(u);                              // write DB first
  redis.del("user:" + u.getId());                // invalidate cache
}

Pattern 2 — Read-Through

Same shape as Cache-Aside but with the cache (or a cache library) doing the DB load itself instead of the app. The app talks only to the cache; the cache transparently fetches from DB on miss.

The "cache" here usually means a library that wraps both — Caffeine's LoadingCache, AWS DAX in front of DynamoDB, or Spring's @Cacheable. Raj (the new backend dev) often confuses Read-Through with Cache-Aside, but the distinction matters: in Read-Through, the cache knows how to load from the DB. In Cache-Aside, only the app does.

@Cacheable(value = "users", key = "#id")
public User getUser(long id) {
  // Spring checks cache first. On miss, this body runs and the result is auto-stored.
  return userRepo.findById(id);
}

Pattern 3 — Write-Through

On a write, the app updates the cache, which synchronously writes to the DB before returning. The cache and the DB are always consistent — but the write is as slow as the slower of the two.

Pattern 4 — Write-Behind (a.k.a. Write-Back)

The app writes only to the cache. The cache asynchronously flushes batched writes to the DB later — every 100 ms, every 1000 writes, whatever the policy says.

This is what high-throughput write systems use — think Cassandra's commit log + memtable, or Kafka before it became its own thing. The user-facing write returns in 1 ms; the durable write to the source-of-truth happens out of band.

Pattern 5 — Write-Around

On a write, skip the cache entirely. Write straight to the DB. Reads still go through the cache (Cache-Aside style).

Why would you ever not populate the cache on write? Because some writes are writes you'll never read. Imagine an audit-log table — billions of inserts, almost no one reads them. Filling the cache with audit-log rows just evicts the data people actually want. Write-Around says "let the cache stay focused on hot reads."

Pattern 6 — Refresh-Ahead

The cache proactively refreshes a key before it expires, based on predicted access. If a key has TTL 60 seconds and is being read every second, the cache might refresh it at the 50-second mark so users never see a miss.

Side-by-side comparison

Eviction policies — when the cache is full, what gets thrown out?

A cache is, by definition, smaller than the data set it's caching. So when it fills up, something has to go. The eviction policy is the rule that picks the victim. The right choice depends on the shape of your access pattern, not on which one sounds smartest.

One cache box isn't enough — going distributed

One Redis box can serve maybe ~100k ops/sec and hold maybe 100 GB of data. At Instagram scale (~500M daily users, billions of cache reads per minute), no single box could possibly cope. So we need to shard the cache across many boxes. And the way you shard determines whether adding a server is a quiet Tuesday or a 3 AM page.

Strategy A — Replication (every node has a copy)

Every cache node holds the entire dataset. On a read, ask any node. On a write, broadcast to all of them.

Pros: any node can answer any query — great read scaling, simple failover. Cons: total capacity = capacity of a single node (replication doesn't add storage, only read throughput), and write amplification grows linearly with replicas. Used by: Redis read replicas, NGINX shared-cache.

Strategy B — Partitioning via consistent hashing

Split the keyspace across nodes. Each key lives on one node only. Lookups need a deterministic rule: "given key K, which node holds it?"

The naive answer — node_index = hash(K) % N — has a brutal failure mode. Add one node and almost every key remaps. Going from 4 to 5 nodes invalidates ~80% of cache entries. A cache that empties every time you scale isn't a cache, it's a heart attack.

The fix is consistent hashing. Imagine a ring labelled 0 to 2^32. Hash each node's name onto the ring; hash each key onto the ring; a key belongs to the first node you encounter walking clockwise from the key's position. Add a node? Only the keys between its position and the next node need to move — typically 1/N of the dataset. That's a controlled flinch instead of a catastrophe.

One more refinement: in practice you don't put each node on the ring once — you put it on ~100–200 virtual nodes, spread evenly. Why? Because with 4 physical nodes you'd get unlucky hash positions and one node would get 40% of traffic while another gets 10%. With 400 virtual nodes the distribution smooths out. This is what Cassandra calls "vnodes" and what every modern hash-partitioned system does.

Hybrid — partitioning + replication

Production systems combine both. Each key lives on one primary shard (partitioned via consistent hashing for capacity), and that shard is replicated to 2 or 3 nodes (for read scaling and failover). Lose a node? The replica takes over. Need more capacity? Add a shard.

Invalidation — the hardest problem in caching

You wrote the new value to the database. Six caches across three data centers still have the old one. How do you tell every one of them? This is the question every senior engineer has tripped on at least once. Below are the five techniques in real use — most production systems combine several.

The five famous cache failure modes

The patterns above all work fine on the happy path. The senior-engineer test is: what happens when traffic spikes, the cache cold-starts, or someone queries a non-existent ID a million times a second? These are the five failures every interviewer wants you to name.

① Thundering Herd (a.k.a. Cache Stampede)

The scene: a popular cache key — say, homepage:top-posts — expires at exactly 11:00:00. In the millisecond after expiry, 10,000 concurrent requests all miss, all hit the database, all compute the same expensive aggregation, and all write it back. The database falls over. The cache is now warm again — for the next 60 seconds, until the next expiry, when the herd thunders again.

public List<Post> topPosts() {
  List<Post> cached = redis.get("top-posts");
  if (cached != null) return cached;

  // Only ONE thread per JVM gets through this lock per key
  synchronized (lockFor("top-posts")) {
    cached = redis.get("top-posts");            // double-check after lock
    if (cached != null) return cached;
    List<Post> fresh = expensiveQuery();
    redis.setex("top-posts", 60, fresh);
    return fresh;
  }
}

For multi-server protection use Redis SETNX as a distributed lock, or better — use a Probabilistic Early Expiration (PEE) scheme: each reader has a small probability of triggering a refresh before the TTL expires, proportional to how close to expiry the key is. By the time the TTL actually hits, the value has already been refreshed by one lucky reader.

② Cache Avalanche (mass expiry)

The scene: at 11:00 AM, after a deploy, the cache fills with thousands of keys all set to TTL = 1 hour. At 12:00 PM, every key expires within the same minute. Suddenly the database is hammered by every key in the system simultaneously.

The fix — jittered TTLs: never set every key to exactly the same TTL. Add a small random jitter — instead of EX 3600, use EX 3600 + rand(0..300). Now the expirations are spread across a 5-minute window instead of a 1-second window. Cheap, effective, and almost no one remembers to do it until they've been bitten.

③ Cache Penetration (queries for things that don't exist)

The scene: an attacker (or a buggy client) queries user:99999999 a million times a second. The cache misses (no such user). The DB returns "not found". The cache doesn't store a "negative" answer. Every single one of those million requests hits the DB.

④ Hot Key (one key, all the traffic)

The scene: Cristiano Ronaldo's Instagram profile gets 50% of all reads to your sharded cache. Whichever shard holds his key gets 50% of all traffic. That shard's CPU spikes to 100% while the other shards yawn at 5%.

The fixes:

• Replicate the hot key — store ronaldo:0, ronaldo:1, …, ronaldo:9 on different shards. Reads pick a random suffix.
• Pin the hot key in an in-process cache on every app server. Network-free reads, but reads are now "eventually consistent" for that key.
• Identify hot keys automatically — Redis 7+ has --hotkeys stats. Tools like Twitter's Pelikan or Netflix's EVCache do this in production.

⑤ Big Key (one value too large)

The scene: someone caches a 100 MB JSON blob under one key. Every read serializes 100 MB across the network. Every write blocks the cache (Redis is single-threaded — a big write stalls every other request). Memory fragments. The eviction policy is forced to dump 99 small useful entries to make room for one giant one.

Fix: split big values. If you must cache a large object, break it into chunks (thread:42:posts:0, thread:42:posts:1, …) or use a Redis hash/list with per-field operations so you can read just the bit you need.

Putting it together — a real cache architecture

Let's walk through how a real production cache evolves, the same way a team would build it. We start from the simplest possible thing, watch it break, fix the break, and repeat until we land on a shape that actually survives a Black Friday.

Pass 1 — The naive design

One service, one Redis box, one Postgres. App talks to Redis on every read, falls back to Postgres on miss. This is what your first Cache-Aside looks like.

Works beautifully at 100 RPS. At 10k RPS, three things break:

• Redis becomes a single point of failure — if it crashes, every read falls through to Postgres, which immediately tips over.
• Redis fills up. 100 GB box; the working set is 200 GB. Eviction thrash.
• One hot key (Ronaldo) saturates the network card on the Redis box.

Pass 2 — The mental model: hot vs warm vs cold

Before scaling anything, name the split. We have three tiers of data, each with very different access shapes — and we should serve them from three different places.

Each tier has its own eviction policy, its own consistency story, and its own monitoring. Hot uses W-TinyLFU with short TTL + pub/sub invalidation. Warm uses LRU + TTL + jitter + explicit invalidation. Cold uses whatever your DB does (buffer pool LRU).

Pass 3 — The production shape

Now we wire it all together. Numbers in the diagram match the cards below.

Component-by-component — what each numbered box does

Use the numbers in the diagram above to find the matching card below. Each card answers four things: what it is, why it exists, what would break without it, and where it sits in the flow.

Concrete walkthrough — Sarah loads her profile page

Let's trace one request through all 10 components. Sarah opens example.com/sarah on her phone in Mumbai. Here's what happens, second by second:

1. Her phone's ① browser cache serves the HTML shell + CSS instantly — those haven't changed.
2. The page's JS calls /api/users/sarah. The request hits ② Cloudflare's Mumbai PoP, which has a 60-second cached copy. Cache hit, served in 5 ms.
3. One second later, Sarah edits her bio. The PUT request bypasses the CDN (write methods aren't cached), goes through ③ the LB to ④ an app server.
4. The app server checks ⑤ the Bloom filter — does user "sarah" definitely-not exist? No, she does, proceed.
5. The app server writes to ⑨ Postgres primary (~10 ms).
6. It then issues DEL user:sarah to ⑥ Redis (single shard via consistent hash).
7. It publishes {"key":"user:sarah"} on ⑧ the pub/sub bus. Within ~50 ms, every other app server's ④ in-process cache evicts its copy of user:sarah.
8. Sarah's next read of her own profile comes back. App server-A checks its in-process cache → miss (just evicted). Checks Redis → miss (we DEL'd it). Goes to ⑨ Postgres → gets the new bio. Stores back into Redis and into in-process cache. Total: ~20 ms.
9. Meanwhile, in parallel, ⑦ the hot-key replicator notices that celebrity:ronaldo is being read 200k times/sec on shard-3 and starts mirroring it to shards 4 and 5 under suffixed keys. Shard-3 CPU drops from 92% to 34%.
10. ⑩ Metrics records every hop's latency. The Grafana dashboard shows the cache hit ratio holding at 96.4% — exactly where it should be.

Picking the right strategy — a cheat sheet

The hardest part of cache design isn't knowing the patterns — it's knowing which to pick for the workload in front of you. The table below maps common scenarios to their best default. Use it as a starting point, not a law.

Interview Q&A — the 15 most-asked questions

These are the questions that come up in distributed-systems rounds — caching turns up in nearly every system-design interview. Each answer is short enough to deliver in 60 seconds while still showing depth.

The one-page summary

If you remember nothing else from this article, remember these.

• Layers: Browser → CDN → Gateway → In-process → Distributed cache → DB. Each protects the one behind it.
• Default pattern: Cache-Aside with TTL + explicit invalidation on write. Reach for others only when this doesn't fit.
• Default eviction: TTL + LRU (or W-TinyLFU via Caffeine for in-process JVM caches).
• Distributed: consistent hashing + virtual nodes + replication factor ≥ 2. Never plain modulo.
• Invalidation: never rely on one mechanism. Combine TTL (safety net) + explicit DEL (common case) + pub/sub (in-process fan-out).
• Always jitter TTLs. Mass simultaneous expiry is one query away from a Sev1.
• Mitigate stampede via single-flight, PEE, or stale-while-revalidate.
• Mitigate penetration via negative caching + Bloom filter.
• Mitigate hot keys by replicating with suffixed keys, or pinning in-process.
• The DB must always survive a cache that misbehaves. If it can't, your cache was load-bearing — an architecture smell.
• Measure hit ratio per layer. Without numbers you're flying blind.
• Caching is an optimization, not a contract. Treat it like one.