When streaming with openai-python (v1.55+), every chunk except the last one has chunk.usage set to None. Trying to access chunk.usage.total_tokens or chunk.usage.prompt_tokens mid-stream throws AttributeError: 'NoneType' object has no attribute 'total_tokens'.
I know the server auto-injects stream_options.include_usage=True (saw it in app/routes/chat.py), so usage IS present — but only on the final chunk before [DONE]. The problem is that OpenAI's own API also puts usage only on the final chunk, but their SDK gracefully handles the None case. When code is written against OpenAI directly and then pointed at OrcaRouter, patterns like this break:
stream = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Explain quantum computing"}],
stream=True,
)
total_tokens = 0
for chunk in stream:
if chunk.usage: # This guard works with openai SDK against OpenAI
total_tokens += chunk.usage.total_tokens
# But this crashes:
# total_tokens = chunk.usage.total_tokens # NoneType error on all but last chunk
for choice in chunk.choices:
print(choice.delta.content or "", end="")
The bigger issue is that with model="auto", I don't know which model actually served the request until I read the stream. The x-orca-resolved-model header is set before the first chunk, which is great, but the usage data I need for billing my own users only arrives at the very end. If the client disconnects mid-stream (user closes the tab), the usage is lost entirely from the client's perspective.
Two possible improvements:
- Carry a cumulative
usage on every chunk (incremental prompt_tokens visible from chunk 1, completion_tokens growing with each delta).
- Or at minimum, document clearly that
chunk.usage is None until the terminal chunk and show the guard pattern in the README streaming section.
I'd prefer option 1 — it makes mid-stream cost tracking possible without waiting for stream completion.
When streaming with
openai-python(v1.55+), every chunk except the last one haschunk.usageset toNone. Trying to accesschunk.usage.total_tokensorchunk.usage.prompt_tokensmid-stream throwsAttributeError: 'NoneType' object has no attribute 'total_tokens'.I know the server auto-injects
stream_options.include_usage=True(saw it inapp/routes/chat.py), so usage IS present — but only on the final chunk before[DONE]. The problem is that OpenAI's own API also puts usage only on the final chunk, but their SDK gracefully handles theNonecase. When code is written against OpenAI directly and then pointed at OrcaRouter, patterns like this break:The bigger issue is that with
model="auto", I don't know which model actually served the request until I read the stream. Thex-orca-resolved-modelheader is set before the first chunk, which is great, but the usage data I need for billing my own users only arrives at the very end. If the client disconnects mid-stream (user closes the tab), the usage is lost entirely from the client's perspective.Two possible improvements:
usageon every chunk (incremental prompt_tokens visible from chunk 1, completion_tokens growing with each delta).chunk.usageisNoneuntil the terminal chunk and show the guard pattern in the README streaming section.I'd prefer option 1 — it makes mid-stream cost tracking possible without waiting for stream completion.