Skip to content

Streaming chunks have usage: null until the final chunk — SDK attribute access crashes #125

Description

@interview35

When streaming with openai-python (v1.55+), every chunk except the last one has chunk.usage set to None. Trying to access chunk.usage.total_tokens or chunk.usage.prompt_tokens mid-stream throws AttributeError: 'NoneType' object has no attribute 'total_tokens'.

I know the server auto-injects stream_options.include_usage=True (saw it in app/routes/chat.py), so usage IS present — but only on the final chunk before [DONE]. The problem is that OpenAI's own API also puts usage only on the final chunk, but their SDK gracefully handles the None case. When code is written against OpenAI directly and then pointed at OrcaRouter, patterns like this break:

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Explain quantum computing"}],
    stream=True,
)

total_tokens = 0
for chunk in stream:
    if chunk.usage:  # This guard works with openai SDK against OpenAI
        total_tokens += chunk.usage.total_tokens
    # But this crashes:
    # total_tokens = chunk.usage.total_tokens  # NoneType error on all but last chunk
    for choice in chunk.choices:
        print(choice.delta.content or "", end="")

The bigger issue is that with model="auto", I don't know which model actually served the request until I read the stream. The x-orca-resolved-model header is set before the first chunk, which is great, but the usage data I need for billing my own users only arrives at the very end. If the client disconnects mid-stream (user closes the tab), the usage is lost entirely from the client's perspective.

Two possible improvements:

  1. Carry a cumulative usage on every chunk (incremental prompt_tokens visible from chunk 1, completion_tokens growing with each delta).
  2. Or at minimum, document clearly that chunk.usage is None until the terminal chunk and show the guard pattern in the README streaming section.

I'd prefer option 1 — it makes mid-stream cost tracking possible without waiting for stream completion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions