Node.js Fundamentals: The Runtime, the Event Loop, and Building Real APIs

Working with the File System & Streams

The problem with fs.readFile on large files

const fs = require("fs");

fs.readFile("./access.log", (err, data) => {
  // `data` is the ENTIRE file's contents, held in memory as one Buffer, at once
  res.end(data);
});

fs.readFile (and its Promise-based sibling) waits for the entire file to be read into memory before your callback runs at all. For a 10KB config file, that's irrelevant. For a 4GB log file, you're allocating gigabytes of process memory just to serve it, the client waits for the entire read to finish before receiving a single byte, and doing this concurrently for a handful of requests can exhaust available memory outright and crash the process.

Streams as the fix

A stream processes data in chunks as it arrives, instead of waiting for the whole thing:

const fs = require("fs");
const http = require("http");

http.createServer((req, res) => {
  const readStream = fs.createReadStream("./access.log"); // reads in chunks, not all at once
  readStream.pipe(res); // writes each chunk to the response as it's read
}).listen(3000);

.pipe() connects a readable stream (the file) to a writable stream (the HTTP response), forwarding each chunk as it becomes available. The client starts receiving bytes almost immediately, and memory usage stays roughly constant regardless of file size — you're only ever holding one chunk (a few dozen KB, by default) in memory at a time, not the whole file.

Backpressure

Piping isn't just "copy data from A to B as fast as possible" — it has to account for backpressure: what happens when the readable side produces data faster than the writable side can consume it (a fast local disk feeding a slow network connection, for instance). A writable stream's .write() call returns false when its internal buffer has filled past a threshold, signaling "stop sending me data for a moment." The stream then emits a 'drain' event once it's caught up and ready for more.

readStream.on("data", (chunk) => {
  const canContinue = writeStream.write(chunk);
  if (!canContinue) {
    readStream.pause(); // stop reading until the writable side catches up
    writeStream.once("drain", () => readStream.resume());
  }
});

That's exactly the bookkeeping .pipe() does for you automatically — it's the main reason to prefer .pipe() (or the stream.pipeline() helper, which also handles error propagation and cleanup correctly) over manually wiring 'data' and 'write' events yourself. Skip backpressure handling in a hand-rolled stream pipeline, and a slow consumer paired with a fast producer will just keep buffering in memory until the process runs out of it.