Chapter 8 covered the algorithms — token bucket, leaky bucket, window counters — as if a single process were enforcing them. The actual system design question is different and more interesting: where does the counter state live once there's more than one application server enforcing the same limit?
If each application server keeps its own in-memory counter per client, and there are five app servers behind a load balancer, a client limited to 100 requests/minute can actually make up to 500/minute — 100 against each server — because none of the five counters know about the other four. This is the exact statelessness tension from chapter 2 in a new costume: the rate limit is state, and state that lives independently on each instance stops being a correctly enforced shared limit the moment there's more than one instance.
The fix is to move the counter into a shared, low-latency store every app server reads from and writes to — Redis is the overwhelmingly common real-world choice, specifically because it's fast enough to sit on the hot path of every request and supports the atomic operations this needs:
Client -> Load Balancer -> App Server (any of N) -> Redis (shared counter, keyed by client ID)
|
Database (unrelated to rate limiting)
Now every app server checks and increments the same counter for a given client, regardless of which server happens to receive that client's request — which restores the correctness the naive in-memory version was missing.
Centralizing the counter isn't automatically enough on its own. A naive implementation — read the current count, check it against the limit, then write back the incremented count as three separate operations — has a race condition the moment two app servers handle two requests from the same client at nearly the same instant: both read the same starting count, both see it's under the limit, both increment, and the client just got two requests through on what should have been the very last available slot. This is a classic check-then-act race, structurally identical to two threads both reading a shared counter before either one's write lands.
The fix is to make the check-and-increment a single atomic operation instead of three separate steps — Redis supports this natively (an atomic INCR, or a small Lua script executed atomically for the token-bucket/sliding-window math), so the read, the limit check, and the write all happen as one indivisible unit no second request can interleave with:
-- Executed atomically (e.g. as a Redis Lua script) --
count = INCR(client_key)
if count == 1:
EXPIRE(client_key, window_size) -- start the window on the first request
if count > limit:
return DENY
return ALLOW
A shared Redis-backed counter means every single request now pays the latency of a network round trip to Redis before the app server can even begin real work — usually on the order of a millisecond or so, small but not zero, and now sitting on the critical path of every request the system serves. Some designs relax this deliberately: each app server keeps a local approximate counter and only reconciles with the shared store periodically, trading perfectly precise global enforcement (a client might briefly exceed the limit by a small, bounded amount across the fleet) for lower per-request latency. Whether that trade is acceptable is exactly the kind of judgment call this reference has been building toward — a payment-fraud rate limiter probably wants the strict, atomic, centralized version despite the latency cost; a "don't hammer our own internal logging endpoint too hard" limiter can probably live with an approximate one.
Distributed rate limiting also inherits a subtler problem worth naming in an interview even briefly: clock skew across the servers involved in window-based algorithms (fixed or sliding window) can cause a client near a window boundary to be judged slightly differently depending on which server's clock it's compared against — one more reason a centralized store with one authoritative clock (Redis's own server time, for instance, rather than each app server's local clock) tends to be more correct than a fully distributed calculation trusting every app server's own notion of "now."