Response Streaming
Suppose we have built a chat application. A user types a question, presses Enter, and our server dutifully forwards it to Claude. Claude begins composing a response. The process can take ten, twenty, thirty seconds.
And what does our user see during those thirty seconds? A spinner.
They check their phone. They wonder if the app is broken. They consider, briefly, a life without our product.
The response, when it finally arrives, is excellent. But it does not matter. The damage is done.
The Problem with Standard Responses #
In the standard setup, our server sends the user’s message to Claude and then waits for the complete response before sending anything back to the client. Nothing escapes until the last word is written.
The user receives no feedback that anything is happening. Thirty seconds of silence, then a wall of text. We want to let them watch the answer being steamed in instead.
How Streaming Works #
With streaming enabled, Claude immediately sends back an initial response saying, in effect, “message received, generation underway.” Then it sends a series of events, each carrying a small piece of the overall response.
Our server can forward these text chunks to the client as they arrive, and the user watches the answer build up word by word. And note well: all of these events belong to a single request to Claude — one request, many deliveries.
Understanding Stream Events #
What arrives on this stream? Several types of events, each announcing itself with a type field:
message_start— a new message is being sentcontent_block_start— the start of a new block containing text, tool use, or other contentcontent_block_delta— chunks of the actual generated textcontent_block_stop— the current content block has been completedmessage_delta— the current message is completemessage_stop— the end of information about the current message
Of these, content_block_delta carries the treasure: the actual generated text we want to show our users.
Basic Streaming Implementation #
To enable streaming, we add stream: true to our messages.create call. The return value is no longer a message — it is an async iterable of events.
const messages: Anthropic.MessageParam[] = [];
addUserMessage(messages, "Write a 1 sentence description of a fake database");
const events = await client.messages.create({
model,
max_tokens: 1000,
messages,
stream: true,
});
for await (const event of events) {
console.log(event);
}
Run this, and behold: a torrent of event objects. Starts, deltas, stops, metadata — the full ceremony, in exhaustive JSON. Somewhere in that flood is the sentence we asked for, diced into fragments.
We could fish the text out ourselves, checking each event’s type and plucking the deltas.
But must we?
Simplified Text Streaming #
The Anthropic SDK provides a simplified streaming interface: client.messages.stream. Instead of parsing events by hand, we subscribe to its text event, which fires with each chunk of generated text and nothing else.
const stream = client.messages.stream({
model,
max_tokens: 1000,
messages,
});
stream.on("text", (text) => {
process.stdout.write(text);
});
Each callback receives pure text, which is usually all we need for displaying responses to users. The sentence assembles itself on screen, word by word.
Getting the Complete Message #
One loose end. Streaming chunks delights the user, but our application often needs the complete message afterwards. Perhaps to store in a database, append to the conversation history, or process further.
The stream keeps assembling the full message behind the scenes, and finalMessage() hands it over once generation completes:
const stream = client.messages.stream({
model,
max_tokens: 1000,
messages,
});
stream.on("text", (text) => {
// Send each chunk to your client
process.stdout.write(text);
});
const finalMessage = await stream.finalMessage();
// A complete message object, ready for database storage
Real-time streaming for the user, a complete message object for our application logic.
Both from one request.
Repo #
Code exercises set up here