What you'll learn

By the end of this tutorial you'll understand what Node.js streams are and why they exist, how Readable, Writable, and Transform streams work, how to compose them with pipe(), and how to handle errors in a stream pipeline without the whole thing crashing silently. You'll also have two working examples — a file processor and a data transformation pipeline — that demonstrate the concepts in a way you can adapt for real use.

Streams are one of those Node.js topics that feels abstract until it clicks, at which point you wonder how you ever processed large files without them.

Who this is for

  • Node.js developers who've read about streams but haven't written one from scratch
  • Anyone who's processed a large file by reading it all into memory and hit a performance or memory wall
  • Developers building data pipelines, file processors, or any feature that involves moving data from one place to another in chunks

You can skip this if you're already building Transform streams and piping them correctly, handling backpressure, and writing async stream generators with Readable.from(). Come back if you need to revisit the fundamentals or debug a stream pipeline.

What are Node.js streams?

Streams are objects that let you read or write data in chunks, rather than waiting for the entire data set to be available. They're the Node.js abstraction for I/O — file reads, HTTP requests, database results, any operation that produces or consumes data over time.

Plain English: instead of reading a 1 GB file entirely into memory and then processing it, a stream reads a small chunk, processes it, moves on to the next chunk, and so on. The memory footprint stays small regardless of how big the total data is.

Simple idea: think of a stream like a conveyor belt. Items come out one at a time, get processed, and move on. You don't wait for all items to be on the belt before you start processing — you handle each one as it arrives.

Prerequisites

  • Node.js 20 installed (node -v)
  • Comfortable with async/await and callbacks in Node.js
  • Basic file system operations (fs module)

No packages needed — streams are a core part of Node.js with no installation required.

Setup from zero

Step 1 — Create a test file

mkdir streams-lab && cd streams-lab
touch main.js

# Create a sample file to stream
node -e "require('fs').writeFileSync('data.txt', 'hello\n'.repeat(10000))"

Step 2 — Read a file the non-stream way and observe memory use

// main.js
const fs = require("fs");

const data = fs.readFileSync("data.txt", "utf8");
console.log("File length:", data.length, "bytes");
console.log("Memory:", process.memoryUsage().heapUsed, "bytes");

With a small test file this runs fine. Replace the file with something large — 500 MB, a real log file — and this blocks the process until the entire file is loaded into memory. Streams fix this.

Step 3 — Read the same file with a stream

const fs = require("fs");

const stream = fs.createReadStream("data.txt", { encoding: "utf8" });
let count = 0;

stream.on("data", (chunk) => {
  count += chunk.split("\n").length - 1;
});

stream.on("end", () => {
  console.log("Line count:", count);
  console.log("Memory:", process.memoryUsage().heapUsed, "bytes");
});

stream.on("error", (err) => {
  console.error("Stream error:", err.message);
});

The data event fires repeatedly with each chunk. The end event fires when there's no more data. The file is never loaded entirely into memory.

> Little tip: the chunk size on a file stream defaults to 64 KB. You can control it with the highWaterMark option: fs.createReadStream("file.txt", { highWaterMark: 16 * 1024 }). Smaller chunks mean more events but lower memory use per chunk; larger chunks are fewer events but more memory at once.

The mental model

Node.js has four stream types:

Readable — a source of data. The file read stream is a Readable. An HTTP request body is a Readable. A database cursor can be a Readable.

Writable — a destination for data. A file write stream is a Writable. An HTTP response is a Writable.

Duplex — both Readable and Writable at once. A TCP socket is a Duplex — data flows in and out.

Transform — a Duplex where the data is modified in transit. A gzip compressor is a Transform. A CSV parser is a Transform. You read data in, transform it, and write modified data out.

The most important concept is backpressure: when a Readable is producing data faster than a Writable can consume it, the data piles up. The pipe() method handles backpressure automatically — it pauses the Readable when the Writable's internal buffer is full and resumes it when space is available.

Understanding this is what separates correct stream pipelines from ones that silently drop data or exhaust memory.

Key terms

Readable — a stream you can read from. Emits data, end, and error events.

Writable — a stream you can write to. Has a write(chunk) method and emits finish and error events.

Transform — a Readable/Writable pair where input is transformed into output. Extends stream.Transform; implement _transform(chunk, encoding, callback).

pipe() — a method on a Readable that connects it to a Writable, handling data flow and backpressure automatically. Superseded in newer code by stream.pipeline() which also handles errors.

pipeline() — a stream module utility that pipes streams together and calls a callback with any error from any stage. The correct modern alternative to chaining .pipe() calls.

Backpressure — the condition where a consumer can't keep up with a producer. pipeline() and pipe() handle this; manually writing to streams without checking the return value of write() can cause memory overflow.

highWaterMark — the internal buffer size of a stream. When the buffer is full, the stream pauses reading until it drains.

Step-by-step

Write a Transform stream

const { Transform } = require("stream");

// Uppercases all text passing through it
class UpperCaseTransform extends Transform {
  _transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback(); // signal that this chunk is done
  }
}

module.exports = UpperCaseTransform;

Build a pipeline with pipeline()

const fs = require("fs");
const { pipeline } = require("stream");
const { createGzip } = require("zlib");
const UpperCaseTransform = require("./UpperCaseTransform");

pipeline(
  fs.createReadStream("data.txt"),
  new UpperCaseTransform(),
  createGzip(),
  fs.createWriteStream("data-upper.txt.gz"),
  (err) => {
    if (err) {
      console.error("Pipeline failed:", err.message);
    } else {
      console.log("Pipeline complete");
    }
  }
);

The data flows: file read → uppercase transform → gzip compress → file write. Each stage processes one chunk at a time. The whole file is never in memory at once.

Working examples

The async pipeline alternative

const { pipeline } = require("stream/promises");
const fs = require("fs");
const { createGzip } = require("zlib");

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    createGzip(),
    fs.createWriteStream(output)
  );
  console.log(`Compressed ${input} → ${output}`);
}

compressFile("data.txt", "data.txt.gz").catch(console.error);

stream/promises gives you a Promise-based pipeline that works cleanly with async/await. Error handling is just try/catch.

Creating a Readable from an async generator

const { Readable } = require("stream");

async function* generateNumbers(count) {
  for (let i = 1; i <= count; i++) {
    yield `${i}\n`;
    await new Promise(r => setTimeout(r, 1)); // simulate async source
  }
}

const source = Readable.from(generateNumbers(100));
source.pipe(process.stdout);

Readable.from() turns any async iterable into a Readable stream. This is the cleanest way to create a custom data source in modern Node.js — no manual _read() implementation needed.

> Little tip: always use stream/promises pipeline instead of manually chaining .pipe() calls in production code. Manual .pipe() chains don't propagate errors from intermediate stages — if the gzip transform throws, the file write stream may not close properly. pipeline() cleans up all streams on error, which prevents file descriptor leaks.

Patterns

Stream from a database cursor — many database drivers expose query results as streams or async iterables. Streaming a large query result through a Transform that formats it as CSV, then piping to an HTTP response, is a clean pattern for large data exports with minimal memory use.

Abort a pipeline on a condition — call destroy(new Error("reason")) on any stream in the pipeline to abort the whole chain. pipeline() will call its callback with the error and close all other streams.

Reuse Transform logic — write each transformation as a separate, composable Transform class. An uppercase transform, a filter transform, and a line counter transform can be piped in any order. Small, single-purpose transforms are easier to test and combine than one large monolithic transform.

Common mistakes

Chaining .pipe() without error handling — readable.pipe(transform).pipe(writable) does not propagate errors from transform or writable back to the Readable. If transform errors, the pipeline stalls silently. Always use pipeline() from stream/promises.

Forgetting to call callback() in _transform — the Transform stream won't process the next chunk until you call callback() in _transform. Omitting it silently stalls the stream — data stops flowing and the pipeline appears hung.

Not handling the 'error' event — if you attach a data listener but not an error listener, an unhandled stream error becomes an uncaught exception that crashes the process. Always attach an error handler, or use pipeline() which handles it for you.

Troubleshooting

Stream appears to hang with no data flowing — check that _transform calls callback() after processing each chunk. Also check that a Readable in paused mode has had .resume() called or a data listener attached to switch it to flowing mode.

Pipeline completes but output file is empty — the write stream may have received an end event before data arrived. Confirm the pipeline is constructed in the correct order and that each Transform is passing data through with this.push().

Memory usage grows continuously while streaming — backpressure is not being respected. If you're manually writing to a Writable without checking whether write() returned false, you're buffering data ahead of consumption. Use pipeline() which handles this automatically.

Checklist

  • [ ] File and network I/O that produces large data uses streams rather than reading everything into memory
  • [ ] All stream pipelines use pipeline() from stream/promises, not raw .pipe() chains
  • [ ] Every custom Transform calls callback() after processing each chunk
  • [ ] An error handler is attached to every stream not managed by pipeline()
  • [ ] No unbounded accumulation of chunks in a manual data event handler before processing

Practice task

Build a log file processor: create a Transform stream that reads lines from a text file, filters for lines that contain the word "error" (case-insensitive), and writes the matching lines to a new output file. Use pipeline() from stream/promises. Count the matching lines in a separate Transform. Log the count when the pipeline finishes. Test with a file large enough that you'd notice the difference if you were loading it all into memory first.

FAQ

When should I use streams vs. just loading the whole file?

If the data is small and bounded — a configuration file, a JSON payload under a few MB — loading it fully is fine and simpler. Use streams when the data size is large, unknown, or potentially unbounded: log files, exports, uploads, real-time feeds. The rule of thumb is: if it could ever exceed available memory on the server, use a stream.

Do I need to call readable.destroy() after a pipeline finishes?

Not when using pipeline() — it handles cleanup. If you're using pipe() manually, you should call destroy() on all streams in the chain after the last one emits finish or end, especially if they hold file descriptors or network connections.

Can I use streams with async/await throughout?

Yes. Use stream/promises's pipeline() for the pipeline itself, Readable.from() to turn async iterables into Readable streams, and for await...of to consume a Readable as an async iterable when you need to process each chunk imperatively.

What to learn next

After streams: Node.js child processes (spawning and streaming data between processes), HTTP streaming responses with Express (sending large datasets to clients without buffering), and the node:readline module for line-by-line text processing that builds on Readable streams.

  • Node.js async patterns — the async fundamentals streams are built on
  • Node.js error handling — error propagation in stream pipelines
  • Node.js testing with node:test — testing Transform streams

Takeaways

Streams process data in chunks, keeping memory use constant regardless of data size. There are four types: Readable (source), Writable (destination), Duplex (both), and Transform (both, with modification). Use pipeline() from stream/promises to compose them — it handles backpressure and error propagation automatically. Use Readable.from() to create Readable streams from async iterables cleanly.

If you remember only one thing: use stream/promises's pipeline() — not chained .pipe() calls. Manual pipe chains don't clean up streams on error, which leads to file descriptor leaks and hangs that are very hard to track down in production.