Repository navigation
[Streaming] Token usage not returned when stream is aborted mid-generation #539
Description
Summary
When using streaming with stream_options: { include_usage: true },
the token usage chunk is never received if the stream is aborted
mid-generation. This makes accurate token tracking impossible for
apps that allow users to cancel responses early.
Current Behavior
The usage chunk is emitted as the very last SSE event, after generation
fully completes:
data: {"choices": [{"delta": {"content": "..."}}]} ← content chunks
data: {"choices": [{"finish_reason": "stop"}]} ← generation done
data: {"usage": {"prompt_tokens": 10, ...}} ← never received on abort
data: [DONE]
If the client aborts at any point before generation finishes, the usage
chunk is never sent.
Expected Behavior
One of the following:
Option A — Emit a partial usage chunk when server detects client disconnect:
data: {"usage": {"prompt_tokens": 42, "completion_tokens": 17, "partial": true}}
Option B — Include a running completion_tokens_so_far counter
on every chunk (opt-in via stream_options):
data: {"delta": {"content": "..."}, "usage": {"completion_tokens_so_far": 5}}
Why It Matters
- Billing per user becomes inaccurate when generation is cancelled
- Common real-world patterns that trigger this: user closes tab,
presses stop button, request timeout fires - Workarounds like client-side tiktoken counting are approximate
and inconsistent across models
Reproduction
const stream = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Write a very long essay" }],
stream: true,
stream_options: { include_usage: true },
});
for await (const chunk of stream) {
// abort after first chunk
stream.controller.abort();
break;
}
// usage is never receivedWorkaround Today
Client-side token estimation using js-tiktoken — inaccurate,
especially for tokens with special characters or multi-byte sequences.
Activity
Hovo-Dev commentedon Apr 5, 2026
@rattrayalex @schnerd
markstuart-oai commentedon Sep 26, 2026
Thanks for the clear example, @Hovo-Dev. We reproduced the missing usage after interrupting a stream. Final usage is currently sent at the end, which leaves a real gap for apps that let users cancel early.
Once the client closes the connection, another chunk can't reliably reach it. Opt-in running counts could help, but the last received count might still miss work done before cancellation takes effect. Being able to retrieve final usage afterward would address that gap more directly.
I'm keeping this open as an enhancement. Thanks for explaining the use case and suggesting concrete options.
Hovo-Dev commentedon Sep 26, 2026
Thanks @markstuart-oai, glad you could reproduce it. Agreed, being able to retrieve final usage afterward is the cleanest fix, since it also covers tokens generated after the client aborts.
One idea: the completion id already arrives in the first chunk, so clients have it before any abort. Could GET /chat/completions/{id} return the final usage for aborted streams, ideally without requiring store: true (even a short usage-only retention would be enough for billing)?
Also, quick question: today, if a stream with store: true is aborted, does it get saved with its final usage? If so, that could work as a temporary workaround.
Happy to test anything when it's ready.
hossiendehghan989 commentedon Oct 2, 2026
The important contract gap here is not just that the final SSE chunk can be missed; cancellation currently leaves the client unable to distinguish “generation was interrupted and usage was finalized” from “the request is still running / the connection failed before finalization.” That makes local token accounting inherently non-authoritative.
A robust API could separate delivery from finalization:
response_id(or request identifier) early in the stream.final,partial, orunavailable; do not make a missing chunk imply zero usage.For streaming clients, an optional running counter is still useful for UI, but it should be documented as a lower bound until finalization. The final usage record should be defined as the source of truth for billing and observability, including prompt tokens, cached tokens, completion tokens, and any reasoning/tool-call tokens.
The regression matrix should abort before the first token, during normal output, immediately after a finish event, and after the TCP connection is closed. Each case should verify that the same response ID eventually returns one stable final usage record, while a genuinely failed request remains distinguishable from a finalized cancellation. I can help turn this into an API-contract test plan.