Skip to content

[AISAGE-757] fix(ai-gateway): bill message_start input tokens on early stream end - #8551

Merged
chrarnoldus merged 1 commit into
mainfrom
fix/messages-stream-start-usage
Oct 22, 2026
Merged

chrarnoldus merged 1 commit into
mainfrom
fix/messages-stream-start-usage

Conversation

@chrarnoldus

@chrarnoldus chrarnoldus commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Messages API streams only recorded usage from message_delta. A stream that ended before message_delta arrived (client disconnect, upstream disconnect, or the gateway duration budget) was recorded with 0 tokens. That includes a client that stops reading at content_block_stop, after it already has the full response. OpenRouter and Vercel recover cost through the generation lookup; providers without one, such as Martian for the stealth models, were billed $0.

message_start already reports the input and cache token counts. The parser now records them and merges them with message_delta usage, preferring the message_delta value when it is present. A stream that stops early is billed for at least its prompt. Output tokens still come from message_delta only, so an early stop does not bill the output generated so far.

The "No usage in Messages API stream" warning still fires when message_delta usage is missing.

Verification

  • Local stack with the real Martian upstream: streamed stealth/claude-sonnet-4.6 on /messages and aborted the client at content_block_stop. Before: 0 tokens, $0. After: input tokens recorded and a nonzero cost.

Visual Changes

N/A

Reviewer Notes

@kilo-code-bot

kilo-code-bot Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

The change keeps message_start usage as a fallback when a Messages API stream ends before message_delta, and the three new tests cover early end, omitted delta fields, and delta precedence. I verified the parser and both approval fixtures: the vercel-messages.log.resp.sse fixture has all cache fields in message_delta, so its snapshot is unchanged.

Assumptions: [INFERENCE] Two unproven concerns, neither reproduced: (1) if a provider sent message_delta.usage.input_tokens that already includes cached tokens but omitted the cache fields, the merge would add message_start cache counts on top and over-count input (the intended shape is delta input null, as tested); (2) startUsage retains message_start's output_tokens (typically 1, asserted in the new test), so early-ended streams bill a token or two of output even though the PR description says output comes from message_delta only. Also noted: with the condition now !deltaUsage, the "SUSPICIOUS: No usage in Messages API stream" warning fires for exactly the early-ended streams this PR records usage for, which is stated as intentional but will add Sentry noise on client aborts. No memory-leak risk in the change (one extra per-stream object reference).

Files Reviewed (2 files)
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.ts
  • packages/web-shared/src/lib/ai-gateway/processUsage.messages.test.ts

Reviewed by deepseek-flash-latest · Input: 53.8K · Output: 12.4K · Cached: 598.9K

Review guidance: REVIEW.md from base branch main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants