CodeOath
← All posts
System Design103 min total · 14 parts

System Design Fundamentals for Interviews: Scalability, Trade-offs, and the Framework Interviewers Actually Grade

Part 13 of 14 · ~5 min

Worked Example: Fanline's Waitlist Notifications

A waitlist notifier makes a surprisingly good interview subject precisely because it won't let caching, queues, and rate limiting stay in their own separate boxes — designing it properly forces all three to interact with each other inside a single system, instead of appearing as unconnected chapter topics the way they might on a simpler CRUD project.

Requirements and the Core Tension

Functional: whenever a held seat frees up — a card decline, a hold that simply times out — or fresh inventory opens up, the fans waiting on that show need to hear about it, ideally in the order they joined the list. Non-functional: a notified fan needs a genuine shot at the seat before it's gone again, nobody should be notified twice for the same opening, and the design has to work equally well for a 40-person waitlist behind a Tuesday night at The Locksmith and a waitlist pushing two million people behind Marlow Vance's stadium reunion — which, unsurprisingly, call for almost completely different approaches.

Fan-out on Write vs. Fan-out on Read

Two strategies exist here, and picking between them really just means deciding whether the expense gets paid at send time or at read time:

  • Fan-out on write: the instant a seat opens up, push a notification straight into every eligible fan's own inbox or push queue right then. Reading it back afterward is trivial — a fan's client just opens its own already-built inbox — but telling a huge waitlist about one batch of freed seats means performing that write thousands or millions of times in a single burst.
  • Fan-out on read: publish one lightweight "inventory changed" signal instead, and let each client figure out for itself whether it's relevant the next time it reads. Publishing stays one cheap write regardless of waitlist size, but every single read now carries more work, since relevance has to be computed on the spot rather than looked up.
Notify at send timeNotify at read time
Cost when the seat opensGrows with waitlist size — brutal once it's hugeFlat — one publish no matter the waitlist size
Cost when a fan checksCheap — their inbox is already sitting there builtHeavier — has to be worked out fresh on each read
Fanline's useThe Locksmith's 40-person waitlist, told directly the second a seat opensMarlow Vance's stadium waitlist, where writing to two million inboxes at once would itself trigger the exact stampede the rate limiter exists to stop

Fanline runs a hybrid of the two: notifying at send time covers the ordinary case — nearly every show — and keeps delivery close to instant for a normal-sized list. The read-time fallback only kicks in for the rare waitlist so large that writing a notification to every entry would become the bottleneck by itself, and — worse — would send a wave of fans straight back into the reserve-seat endpoint all at once, exactly the synchronized burst the checkout rate limiter was built to survive rather than trigger. Spelling that interaction out loud — that an eager notification strategy on the very largest waitlists would quietly sabotage an entirely different part of the system — is precisely the kind of connection step five of the framework wants a candidate to catch.

Where Queues and Caching Fit In

Notifying a waitlist isn't one atomic step — it's recording that a seat freed up, deciding who's actually eligible to hear about it, and pushing the notification to whoever's currently connected. Doing all of that inline the moment a hold times out would make that timeout-handling job wait on work it doesn't need to block on, so Fanline instead records the freed seat durably and drops a "seat available" event onto the same pub/sub pattern from the queue chapter — both the fan-out-on-write worker and the stadium-scale fan-out-on-read publisher subscribe to it independently, each handling it in whichever way suits that particular show's waitlist size.

A cache slots in on the read side for the same reason it always does: whether a given fan is still eligible for a notification — has their hold window already closed, did they already get and miss their turn — is a check that fires constantly and hardly ever changes between one check and the next, exactly the pattern cache-aside was made for.

Delivery to Online vs. Offline Fans

A fan with the app open usually has a live connection pinned to one specific server, and that's the server that pushes the notification through the moment it's relevant — which is chapter two's statelessness problem showing up in yet another costume. Knowing which server a given fan is currently connected to is itself a piece of state, and it has to be discoverable from anywhere, typically through a shared lookup table mapping fan to connection-holding server, so that whichever app server generates the notification can track down the one machine actually able to deliver it.

A fan who's nowhere near the app at that moment simply has no open connection to receive anything on — whatever got written to their inbox just sits there untouched, waiting, and a push alert or an email digest eventually gets their attention some other way. Reopening the app later triggers an ordinary read against that same inbox, kept warm by the identical cache-aside pattern this whole reference keeps coming back to, and it surfaces without a single special-cased "handle the offline case" branch anywhere in the code. That's what's quietly elegant about the fan-out-on-write path in particular: a live push and a delayed pickup were never really two different systems. They're one write, sitting in one place, with two different moments someone happens to come read it.