Node.js streams intro

What you'll learn

By the end of this tutorial you'll understand what Node.js streams are and why they exist, how Readable, Writable, and Transform streams work, how to compose them with pipe(), and how to handle errors in a stream pipeline without the whole thing crashing silently. You'll also have two working examples — a file processor and a data transformation pipeline — that demonstrate the concepts in a way you can adapt for real use.

Streams are one of those Node.js topics that feels abstract until it clicks, at which point you wonder how you ever processed large files without them.

Direct answer: use a stream when the input is large, arrives over time, or has no safe upper bound. Reading a whole file first makes memory grow with the file. A correct pipeline keeps only bounded working buffers in flight, so processing can begin before the entire input exists. Streams do not guarantee constant memory if your own code keeps every chunk or ignores backpressure.

Who this is for

• Node.js developers who've read about streams but haven't written one from scratch
• Anyone who's processed a large file by reading it all into memory and hit a performance or memory wall
• Developers building data pipelines, file processors, or any feature that involves moving data from one place to another in chunks

You can skip this if you're already building Transform streams and piping them correctly, handling backpressure, and writing async stream generators with Readable.from(). Come back if you need to revisit the fundamentals or debug a stream pipeline.

What are Node.js streams?

Streams are objects that let you read or write data in chunks, rather than waiting for the entire data set to be available. They're the Node.js abstraction for I/O — file reads, HTTP requests, database results, any operation that produces or consumes data over time.

Plain English: instead of reading a large file entirely into memory and then processing it, a stream reads a chunk, processes it, and moves on. With backpressure and no unbounded accumulation in your code, memory is governed by buffers and in-flight work rather than total file size.

Simple idea: think of a stream like a conveyor belt. Items come out one at a time, get processed, and move on. You don't wait for all items to be on the belt before you start processing — you handle each one as it arrives.

Prerequisites

• A currently supported Node.js release installed (node -v)
• Comfortable with async/await and callbacks in Node.js
• Basic file system operations (fs module)

No packages needed — streams are a core part of Node.js with no installation required.

Setup from zero

Step 1 — Create a test file

mkdir streams-lab && cd streams-lab
touch main.js

# Create a sample file to stream
node -e "require('fs').writeFileSync('data.txt', 'hello\n'.repeat(10000))"

Step 2 — Read a file the non-stream way and observe memory use

// main.js
const fs = require("fs");

const data = fs.readFileSync("data.txt", "utf8");
console.log("File length:", data.length, "bytes");
console.log("Memory:", process.memoryUsage().heapUsed, "bytes");

With a small test file this runs fine. Replace the file with something large — 500 MB, a real log file — and this blocks the process until the entire file is loaded into memory. Streams fix this.

Step 3 — Read the same file with a stream

const fs = require("fs");

const stream = fs.createReadStream("data.txt", { encoding: "utf8" });
let count = 0;

stream.on("data", (chunk) => {
  count += chunk.split("\n").length - 1;
});

stream.on("end", () => {
  console.log("Line count:", count);
  console.log("Memory:", process.memoryUsage().heapUsed, "bytes");
});

stream.on("error", (err) => {
  console.error("Stream error:", err.message);
});

The data event fires repeatedly with each chunk. The end event fires when there's no more data. The file is never loaded entirely into memory.

> Little tip: a file Readable uses a 64 KiB highWaterMark by default. You can change the threshold with fs.createReadStream("file.txt", { highWaterMark: 16 * 1024 }), but it is a buffering threshold, not a universal memory cap. Measure before tuning it.

The mental model

Node.js has four stream types:

Readable — a source of data. The file read stream is a Readable. An HTTP request body is a Readable. A database cursor can be a Readable.

Writable — a destination for data. A file write stream is a Writable. An HTTP response is a Writable.

Duplex — both Readable and Writable at once. A TCP socket is a Duplex — data flows in and out.

Transform — a Duplex where the data is modified in transit. A gzip compressor is a Transform. A CSV parser is a Transform. You read data in, transform it, and write modified data out.

The most important concept is backpressure: when a Readable is producing data faster than a Writable can consume it, the data piles up. The pipe() method handles backpressure automatically — it pauses the Readable when the Writable's internal buffer is full and resumes it when space is available.

Understanding this is what separates correct stream pipelines from ones that silently drop data or exhaust memory.

Key terms

Readable — a stream you can read from. Emits data, end, and error events.

Writable — a stream you can write to. Has a write(chunk) method and emits finish and error events.

Transform — a Readable/Writable pair where input is transformed into output. Extends stream.Transform; implement _transform(chunk, encoding, callback).

pipe() — a method on a Readable that connects it to a Writable and coordinates backpressure. It is still valid; pipeline() is usually safer for a complete multi-stage operation because it centralizes completion, error propagation, and teardown.

pipeline() — a node:stream utility with callback and Promise forms that connects stages, reports completion or failure, and destroys unfinished stages on error.

Backpressure — the condition where a consumer can't keep up with a producer. pipeline() and pipe() handle this; manually writing to streams without checking the return value of write() can cause memory overflow.

highWaterMark — the threshold at which a stream stops asking for more data or write() starts returning false. It is not a strict process-memory limit.

If you write manually instead of using pipe() or pipeline(), respect the Writable's return value. false means stop producing until drain:

const { once } = require("node:events");

for await (const chunk of source) {
  if (!destination.write(chunk)) {
    await once(destination, "drain");
  }
}
destination.end();

Step-by-step

Write a Transform stream

const { Transform } = require("stream");

// Uppercases all text passing through it
class UpperCaseTransform extends Transform {
  _transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback(); // signal that this chunk is done
  }
}

module.exports = UpperCaseTransform;

Build a pipeline with pipeline()

const fs = require("fs");
const { pipeline } = require("stream");
const { createGzip } = require("zlib");
const UpperCaseTransform = require("./UpperCaseTransform");

pipeline(
  fs.createReadStream("data.txt"),
  new UpperCaseTransform(),
  createGzip(),
  fs.createWriteStream("data-upper.txt.gz"),
  (err) => {
    if (err) {
      console.error("Pipeline failed:", err.message);
    } else {
      console.log("Pipeline complete");
    }
  }
);

The data flows: file read → uppercase transform → gzip compress → file write. Each stage processes one chunk at a time. The whole file is never in memory at once.

Working examples

The async pipeline alternative

const { pipeline } = require("stream/promises");
const fs = require("fs");
const { createGzip } = require("zlib");

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    createGzip(),
    fs.createWriteStream(output)
  );
  console.log(`Compressed ${input} → ${output}`);
}

compressFile("data.txt", "data.txt.gz").catch(console.error);

stream/promises gives you a Promise-based pipeline that works cleanly with async/await. Error handling is just try/catch.

Stream a file over HTTP

An HTTP response is a Writable stream, so a download can start before Node has read the whole file:

const http = require("node:http");
const fs = require("node:fs");
const { stat } = require("node:fs/promises");
const { pipeline } = require("node:stream/promises");

http.createServer(async (req, res) => {
  if (req.url !== "/download") {
    res.writeHead(404).end("Not found");
    return;
  }

  let file;
  try {
    file = await stat("report.csv");
  } catch {
    res.writeHead(404).end("File not found");
    return;
  }

  res.writeHead(200, {
    "content-type": "text/csv",
    "content-length": file.size,
    "content-disposition": 'attachment; filename="report.csv"',
  });

  try {
    await pipeline(fs.createReadStream("report.csv"), res);
  } catch (error) {
    console.error("Download failed:", error);
  }
}).listen(3000);

Check errors that can be detected before sending headers first, as the missing-file branch does. Once a pipeline into ServerResponse fails after headers are sent, pipeline() can destroy the socket; you cannot reliably replace that partial response with a JSON error page.

Creating a Readable from an async generator

const { Readable } = require("stream");

async function* generateNumbers(count) {
  for (let i = 1; i <= count; i++) {
    yield `${i}\n`;
    await new Promise(r => setTimeout(r, 1)); // simulate async source
  }
}

const source = Readable.from(generateNumbers(100));
source.pipe(process.stdout);

Readable.from() turns any async iterable into a Readable stream. This is the cleanest way to create a custom data source in modern Node.js — no manual _read() implementation needed.

> Little tip: prefer node:stream/promises pipeline for a complete multi-stage operation. A manual .pipe() chain still needs explicit error and teardown coordination at every stage; pipeline() gives the operation one completion point and destroys unfinished stages on failure.

Patterns

Stream from a database cursor — many database drivers expose query results as streams or async iterables. Streaming a large query result through a Transform that formats it as CSV, then piping to an HTTP response, is a clean pattern for large data exports with minimal memory use.

Abort a pipeline on a condition — pass an AbortSignal to the Promise form, then call controller.abort(). The returned Promise rejects and pipeline() tears down unfinished stages. Direct destroy(error) is still useful when implementing a stream, but cancellation is clearer when one controller owns the operation.

Reuse Transform logic — write each transformation as a separate, composable Transform class. An uppercase transform, a filter transform, and a line counter transform can be piped in any order. Small, single-purpose transforms are easier to test and combine than one large monolithic transform.

Common mistakes

Chaining .pipe() without error handling — readable.pipe(transform).pipe(writable) does not turn all stages into one error-managed operation. An error can leave other stages open unless you coordinate listeners and teardown. Prefer pipeline() from node:stream/promises when the chain represents one job.

Forgetting to call callback() in _transform — the Transform stream won't process the next chunk until you call callback() in _transform. Omitting it silently stalls the stream — data stops flowing and the pipeline appears hung.

Not handling the 'error' event — if you attach a data listener but not an error listener, an unhandled stream error becomes an uncaught exception that crashes the process. Always attach an error handler, or use pipeline() which handles it for you.

Troubleshooting

Stream appears to hang with no data flowing — check that _transform calls callback() after processing each chunk. Also check that a Readable in paused mode has had .resume() called or a data listener attached to switch it to flowing mode.

Pipeline completes but output file is empty — the write stream may have received an end event before data arrived. Confirm the pipeline is constructed in the correct order and that each Transform is passing data through with this.push().

Memory usage grows continuously while streaming — backpressure is not being respected. If you're manually writing to a Writable without checking whether write() returned false, you're buffering data ahead of consumption. Use pipeline() which handles this automatically.

Checklist

• [ ] File and network I/O that produces large data uses streams rather than reading everything into memory
• [ ] Multi-stage jobs use pipeline() from node:stream/promises or have equivalent explicit error and teardown handling
• [ ] Every custom Transform calls callback() after processing each chunk
• [ ] An error handler is attached to every stream not managed by pipeline()
• [ ] No unbounded accumulation of chunks in a manual data event handler before processing

Practice task

Build a log file processor: create a Transform stream that reads lines from a text file, filters for lines that contain the word "error" (case-insensitive), and writes the matching lines to a new output file. Use pipeline() from stream/promises. Count the matching lines in a separate Transform. Log the count when the pipeline finishes. Test with a file large enough that you'd notice the difference if you were loading it all into memory first.

FAQ

When should I use streams vs. just loading the whole file?

If the data is small and bounded — a configuration file, a JSON payload under a few MB — loading it fully is fine and simpler. Use streams when the data size is large, unknown, or potentially unbounded: log files, exports, uploads, real-time feeds. The rule of thumb is: if it could ever exceed available memory on the server, use a stream.

Do I need to call readable.destroy() after a pipeline finishes?

No. Do not destroy a stream merely because a normal pipeline(), finish, or end has completed. Use destroy(error) to abort or fail unfinished work, and make custom streams release resources in their lifecycle methods. A manual pipe() chain needs coordinated error and cancellation handling, not blanket destruction after success.

Can I use streams with async/await throughout?

Yes. Use stream/promises's pipeline() for the pipeline itself, Readable.from() to turn async iterables into Readable streams, and for await...of to consume a Readable as an async iterable when you need to process each chunk imperatively.

What to learn next

After streams: Node.js child processes (spawning and streaming data between processes), HTTP streaming responses with Express (sending large datasets to clients without buffering), and the node:readline module for line-by-line text processing that builds on Readable streams.

Takeaways

Streams process data in chunks so memory can be bounded by buffers and in-flight work rather than total input size. There are four classic types: Readable, Writable, Duplex, and Transform. Use pipeline() from node:stream/promises to compose a complete operation with backpressure, a single completion result, and coordinated failure handling. Use Readable.from() to adapt async iterables.

If you remember only one thing: use stream/promises's pipeline() — not chained .pipe() calls. Manual pipe chains don't clean up streams on error, which leads to file descriptor leaks and hangs that are very hard to track down in production.