Cache Stampede / Thundering Herd

Question

A cache eviction triggers a surge of requests to the origin, causing overload. How do you diagnose and prevent cache stampede?

Accepted Answer

Cache stampede is a critical issue where a synchronized cache expiration leads to a sudden, massive surge of requests to the origin server, potentially causing overload and service degradation. Diagnosing and preventing this requires a multi-faceted approach. **Diagnosis - Identifying Synchronized TTL Expirations:** The primary cause is multiple cache entries expiring at the exact same time. I'd start by analyzing cache hit/miss logs and origin server load patterns. Correlating spikes in origin traffic with periods where a significant number of cache keys have recently expired would be the key indicator. Monitoring tools can often flag such synchronized expirations. **Prevention - Implementing Jittered TTLs:** To break this synchronization, I would introduce randomness to Time-To-Live (TTL) values. Instead of setting all TTLs to a fixed duration, they would have a slight random variation (e.g., TTL +/- 5% of the base value). This 'jitter' ensures that keys expire at different times, spreading the load on the origin. **Prevention - Request Coalescing and Locking:** Another effective strategy is to implement a locking or coalescing mechanism. When the first request for an expired cache item arrives, it triggers a background refresh. Subsequent requests for the same item within a short window are either held back (locked) or are also directed to the new, refreshing item, preventing multiple concurrent requests to the origin. This is often implemented within the caching layer or application. **Prevention - Stale-While-Revalidate:** I would leverage the 'stale-while-revalidate' pattern. When a cached item is nearing its expiration, the cache can serve the stale content immediately while asynchronously triggering a background request to revalidate and update the cache from the origin. This ensures users continue to receive responses even during cache refreshes. **Prevention - Early Refresh and Adaptive Caching:** For frequently accessed but non-critical data, implementing an 'early refresh' mechanism is beneficial. This involves proactively refreshing cache entries before their TTL expires, especially when usage patterns indicate potential upcoming demand. Additionally, employing adaptive caching strategies that adjust TTLs based on request frequency and origin responsiveness can further mitigate stampedes. Proactive monitoring of cache hit rates, origin server load, and synchronized expiration events is paramount. By employing a combination of jittered TTLs, request coalescing, and patterns like stale-while-revalidate, we can effectively prevent cache stampedes and ensure application resilience.

Cache Stampede / Thundering Herd

Ready to prove your AWS SysOps knowledge?

Designing Backpressure in Reactive Systems

Caching Strategy Deep Dive

Design an E-commerce Checkout & Cart

Design an API Gateway / Edge Layer

Design a Distributed Caching Layer

Edge Computing Meets Edge Caching: Building Real-Time Applications at the Network Edge

Edge Compute with CDNs: Caching, Workers, and Safe Gradual Rollouts

Paginated vs. Bulk API Calls: Performance, Cost, and Reliability Trade‑offs