How Streaming Works

Streaming means sending data in small pieces over time instead of waiting to have all of it and sending it at once. The receiver starts using the data while the rest is still arriving. This is how video plays before it fully downloads and how large datasets flow without huge memory use.

All at once vs streamed

Buffered (all at once):
  build the whole 2 GB file -> then send -> receiver waits a long time

Streamed (in chunks):
  send [chunk] -> [chunk] -> [chunk] -> receiver uses each as it arrives

The streamed version starts fast and uses little memory on both ends.

Chunked transfer

HTTP can stream a response without knowing the total size in advance using chunked transfer encoding.

HTTP/1.1 200 OK
Content-Type: text/plain
Transfer-Encoding: chunked

7\r\n
Hello, \r\n
6\r\n
world!\r\n
0\r\n
\r\n

Each chunk states its size, then its bytes. A zero-length chunk marks the end.

Backpressure

What if the producer is faster than the consumer? Without control, buffers fill and memory explodes. Backpressure is the mechanism where a slow consumer tells the producer to slow down.

Producer ---> [ buffer ] ---> Consumer (slow)
   ^                              |
   |------ "slow down!" ----------|

Good streaming systems pause production until the consumer is ready, keeping memory bounded.

A streaming read in pseudocode

for await (const chunk of source.stream()) {
  await destination.write(chunk); // waiting here applies backpressure
}

Awaiting each write naturally throttles the source.

Advantages

• Low memory: you never hold the whole payload.
• Fast start: the consumer acts on early chunks immediately.
• Handles data of unknown or unbounded size.

Disadvantages

• More complex than buffering everything.
• Error handling mid-stream is tricky.
• Needs backpressure to stay stable under load.

Use cases

• Video and audio: play while downloading (think live or on-demand media).
• Large file transfers and exports.
• Real-time data feeds: logs, metrics, server-sent events.
• Processing big datasets record by record instead of loading all of it.

Reach for streaming whenever the data is large, continuous, or needs to be used as soon as it starts arriving.