Response Streaming

Suppose we have built a chat application. A user types a question, presses Enter, and our server dutifully forwards it to Claude. Claude begins composing a response. The process can take ten, twenty, thirty seconds.

And what does our user see during those thirty seconds? A spinner.

They check their phone. They wonder if the app is broken. They consider, briefly, a life without our product.

The response, when it finally arrives, is excellent. But it does not matter. The damage is done.

The Problem with Standard Responses #

In the standard setup, our server sends the user’s message to Claude and then waits for the complete response before sending anything back to the client. Nothing escapes until the last word is written.

The user receives no feedback that anything is happening. Thirty seconds of silence, then a wall of text. We want to let them watch the answer being steamed in instead.

How Streaming Works #

With streaming enabled, Claude immediately sends back an initial response saying, in effect, “message received, generation underway.” Then it sends a series of events, each carrying a small piece of the overall response.

Our server can forward these text chunks to the client as they arrive, and the user watches the answer build up word by word. And note well: all of these events belong to a single request to Claude — one request, many deliveries.

Understanding Stream Events #

What arrives on this stream? Several types of events, each announcing itself with a type field:

Of these, content_block_delta carries the treasure: the actual generated text we want to show our users.

Basic Streaming Implementation #

To enable streaming, we add stream: true to our messages.create call. The return value is no longer a message — it is an async iterable of events.

const messages: Anthropic.MessageParam[] = [];
addUserMessage(messages, "Write a 1 sentence description of a fake database");

const events = await client.messages.create({
  model,
  max_tokens: 1000,
  messages,
  stream: true,
});

for await (const event of events) {
  console.log(event);
}

Run this, and behold: a torrent of event objects. Starts, deltas, stops, metadata — the full ceremony, in exhaustive JSON. Somewhere in that flood is the sentence we asked for, diced into fragments.

We could fish the text out ourselves, checking each event’s type and plucking the deltas.

But must we?

Simplified Text Streaming #

The Anthropic SDK provides a simplified streaming interface: client.messages.stream. Instead of parsing events by hand, we subscribe to its text event, which fires with each chunk of generated text and nothing else.

const stream = client.messages.stream({
  model,
  max_tokens: 1000,
  messages,
});

stream.on("text", (text) => {
  process.stdout.write(text);
});

Each callback receives pure text, which is usually all we need for displaying responses to users. The sentence assembles itself on screen, word by word.

Getting the Complete Message #

One loose end. Streaming chunks delights the user, but our application often needs the complete message afterwards. Perhaps to store in a database, append to the conversation history, or process further.

The stream keeps assembling the full message behind the scenes, and finalMessage() hands it over once generation completes:

const stream = client.messages.stream({
  model,
  max_tokens: 1000,
  messages,
});

stream.on("text", (text) => {
  // Send each chunk to your client
  process.stdout.write(text);
});

const finalMessage = await stream.finalMessage();
// A complete message object, ready for database storage

Real-time streaming for the user, a complete message object for our application logic.

Both from one request.

Repo #

Code exercises set up here

 
0
Kudos
 
0
Kudos

Now read this

Where the machine ends

Where does one state machine stop and the next begin? What goes in, what stays out, how do machines cooperate, and how do their diagrams stay honest? Machines model behaviours, not apps # The thing being modelled is not “the application”... Continue →