Repository navigation
feat(backend): a partial quota reading must not score better than a complete one #15348
Description
Activity
- changed the title
[-]feat(auto-combo): a partial quota reading must not score better than a complete one[/-][+]feat(backend): a partial quota reading must not score better than a complete one[/+]on Oct 6, 2026 Thanks for the careful write-up, @shannonlowder. We reproduced this on the current release tip: a reading with only
sessionscores exactly the same as one withsession+weeklyat the same value (80 vs 80), so a missing window can only help a provider. We also confirmed that afractionReported: falsewindow is already dropped when the usage payload is converted (genericQuotaFetcher.ts), so it is never read as a real value, but because it just disappears it is indistinguishable from an absent window and the provider is still scored optimistically.We are parking this as a deferred item rather than building it right now, for two reasons. First, it should reuse the mechanism from #15347 (PR #15358 is open and adds the "unreadable" flag plus a soft penalty), and #15358 explicitly leaves partial readings to this issue. Building both at once would conflict on the same scoring functions and risk two competing mechanisms. Second, the "expected window set" has no source in the codebase today, and window names vary by plan tier and, for Antigravity, by requested-model family, so a naive set would demote healthy providers that legitimately have fewer windows.
When #15358 lands we plan to pick this up as a small follow-up that carries "incomplete" through the same soft-penalty path. It would help a lot if you could share your view on these points: (1) should the expected set be learned per connection from the last complete reading, or declared per provider; (2) how should a plan with a genuinely smaller window set (for example no weekly on a free tier) avoid being flagged; (3) should an incomplete reading use the same 0.5 factor as an unreadable one, or something milder since it still carries real information. The issue stays open with your acceptance criteria while we wait for #15358 and your input.
Thanks, @diegosouzapw, and good to see #15358 land:
quotaUnreadable,UNREADABLE_QUOTA_SOFT_DEPRIORITIZE_FACTORandfinitePercentUsedare now onrelease/v3.8.52, so this can reuse them as you planned. We'll leave the implementation to you; our input on your three questions is below, plus some pointers so they aren't lost when this is picked up.Pointers for whoever picks this up. The expected window set has starting points in the code, each with a caveat (as of
release/v3.8.52@3def9ae310):quotaPreflight.ts:registerQuotaWindows/getQuotaWindows/getAllProviderQuotaWindowsis a per-provider window-name registry. Bespoke fetchers declare their windows (codex, bailian, freemodel, grok-web, llmgateway, lyceum, opencode, qwen token plan, v0, xai-oauth).genericQuotaFetcher.ts(~L448) learns them on every uncached success. Its header says "first successful response", but the code re-registers each time. The registry is keyed per provider and overwritten, so the last connection to refresh wins and a free-tier connection can redefine the set for everyone. That is why it can't be used directly as the "expected" set, which matches your concern about naive sets.src/domain/quotaCache.ts:getQuotaCache(connectionId), hydrated fromquota_snapshots, is already per connection. We have not verified whether it retains a stable set or only the latest reading.percentUsedForQuotadropsfractionReported: falsewindows at conversion, as you found. Carrying their names forward (for example anunreportedWindowslist on the snapshot) is the one place to keep that signal.
1. Declared per provider, or learned per connection? Learned per connection. The repo already uses a seed-then-learn shape:
estimatorCalibration.ts(#14931) starts from a default of 1.0, keeps an EMA per (provider, model), requiresMIN_SAMPLES, clamps, rejects outliers, stays in memory so a restart returns to current behavior, and has an env kill switch. The same shape fits here: a window counts as "expected" for a connection only after N readings in a recent span, and a connection with no history is neutral (cold start = today's behavior). Declared sets would be documentation, not a trigger.2. A plan with a genuinely smaller window set? That falls out of (1): a free tier never reports
weekly, so it never learns it as expected. Two things to handle: an age-out, so a plan downgrade stops being flagged, and Antigravity's model-keyed windows, which could be baselined pergetQuotaFetchScopeor left exempt in a first version.3. Same 0.5 as unreadable? No, milder, and ordered unreadable < incomplete < complete at the same reported value, as a single named constant next to the unreadable one. An incomplete reading still carries real windows, but the missing window can hide the decisive signal. On our instance an Ollama Cloud weekly meter was at 31.3% after 21.6% of the week had elapsed (1.45x pace); a session-only reading would have looked fine. We don't yet have a measured 429 rate by reading class (complete / incomplete / unreadable), so we'd rather not suggest a number from intuition; picking it from that correlation would be the better route.
An optional split, only if useful. (a) Windows the upstream explicitly marks
fractionReported: falseare deterministic and need no baseline, so they can't flag a healthy smaller plan. (b) A window that disappears relative to a learned baseline is the heuristic part. One change covering both is fine too: it deploys sooner, while splitting lets each be proven separately. Your call.Hi @shannonlowder! Thanks for the suggestion.
We cataloged this request in the internal backlog. Closing the issue drains the tracker; it does not discard the idea. When the work is picked up, the implementing pull request will reference this issue. Subscribe to repository releases to see what ships.
Sounds good, ping me if you want me to work any other issues through my agents.
Problem / Use Case
Source: discussion #14270, observation 3, confirmed there. Lines are
release/v3.8.52@ 4db3c54.remainingPercentFromQuotaWindows(open-sse/services/antigravityQuotaFamily.ts:80) skips windows that are not objects or whosepercentUsedis not finite (:88-91), then takesmin()over what remains. A provider that normally reports session + weekly + monthly and this time reports only session is scored on the session window alone. A missing window can only make the provider look better.#10095 added
fractionReportedso a window the upstream did not report cannot be read as a real value, andisExhaustedhonours it (src/domain/quotaCache.ts:125). The scoring path does not:fractionReportedis not read anywhere underopen-sse/services/combo/.Proposed Solution
Track the window set a provider is expected to report and treat a reading with an expected window absent as incomplete: distinct from healthy and from exhausted. How an incomplete reading ranks should reuse whatever #15347 decides for an unreadable one, so the two issues share one mechanism. (If #15347 has not landed, this one can carry a minimal version.)
Alternatives Considered
Treating a missing window as 100% used: wrong in the other direction, it would park a provider for a reporting glitch.
Acceptance Criteria
{ windows: { session: {percentUsed: 0.2} } }withweeklyexpected scores below a complete reading of the same session value.fractionReported: falseis not read as a real value.Area
Proxy / Routing
Expected Test Plan
Unit tests in
tests/unit/for the partial/complete/fractionReported:falsefixtures; each fails on the current tip.