CodeOath
← All posts
Architecture & Patterns68 min total · 17 parts

Microservices vs. Monolith: The Trade You're Actually Making

Part 9 of 17 · ~3 min

Observability Across Services

The subscriber who emailed about her missing lettuce is a real problem, and it's the moment Furrow actually learns why observability stops being optional the instant a request starts touching more than one process. Her box order had confirmed successfully. The app had shown lettuce as part of the box. And yet the box that arrived didn't have any — some substitution had happened somewhere between allocation and packing, and nobody could say where, because the only evidence was three separate services' local logs, on three separate machines, with three separate clocks that weren't even in perfect sync with each other.

Three practices turn that kind of mystery from "we'll never know" into "here's exactly what happened," and none of them are needed inside a single process:

  • Correlation IDs — a unique ID generated once, at the API gateway, the moment her original request comes in, and passed unchanged through every service call it triggers afterward. Every log line from every service touching her order gets tagged with it, so "what happened to order 88214" becomes one searchable query instead of manually lining up three logs by eyeballed timestamps.
  • Centralized logging — every service ships its logs to one searchable place instead of leaving them on whatever pod happened to run that instance, which by the time anyone's investigating might already have been scaled down and gone.
  • Distributed tracing — goes further than a correlation ID: a tracing system (Furrow's runs on OpenTelemetry) records how long each hop in the chain took and which call kicked off which, stitching all of it into one trace built from a trace ID for the request as a whole and a span ID per individual hop — so instead of just knowing Harvest, the substitution logic, and Fulfillment were all touched, you can see exactly how long each one held the request and who handed it to whom next.

With the trace pulled up, the actual answer took four minutes to find: the substitution logic had correctly swapped out lettuce for kale because Harvest's real-time count dipped below the reserved amount during packing — a legitimate, working feature — but nothing in that path updated the packing-list line item the app had already shown the subscriber, so her screen kept saying lettuce right up until the box left the warehouse. Not a bug in any one service. A gap between two services that were each individually correct, which is precisely the kind of failure a single stack trace could never have shown, because no single process ever held the whole story.

Common mistake: bolting on distributed tracing only after the incident that actually needed it, instead of before. Furrow shipped its trace ID plumbing the same week Harvest became its own service — not because anyone foresaw the lettuce mystery specifically, but because being able to say only "somewhere in there, across four services" about a customer's complaint was never going to be a usable answer, and the team knew that going in rather than learning it the hard way.