Four observations on the shape of the quota signal in auto-combo scoring

[shannonlowder]

Four observations on the shape of the quota signal in auto-combo scoring

First: this is offered as discussion, not as a defect report. I run OmniRoute self-hosted against ~840k requests/month across ~10 providers, and I went looking for why my upstream_429 rate climbed from ~0% to 36% over three weeks. That took me through the quota scoring path, and I came away with four observations that seem related — they're all about the shape of the quota signal rather than whether it's wired up.

I want to be explicit that the wiring is already done and I'm not re-reporting it: #4540/#4990 fixed exhausted providers scoring identically to healthy ones, #11411 brought genuine quota-aware routing to generic providers, #12317 declared observed reliability as a factor, and #5923 extended the quota cutoff to the 16 non-auto strategies. #12384 already sketches where auto/* is heading. These four are a layer above that.

All line references are main @ 02c663cd.

---

1. Window length never reaches the score

calculateResetWindowAffinity (combo/quotaScoring.ts:417) scores reset proximity:

return clamp01(1 - msUntilReset / getResetWindowHorizonMs(config.windows));

quota and resetWindowAffinity then enter calculateScore as independent weighted terms. Because they're summed linearly, the scorer can express "45% remaining" and "resets soon", but not the thing that actually predicts exhaustion: usage measured against how much of the window has elapsed.

"45% used of a 5-hour window" and "45% used of a monthly window" are the same number and opposite situations. Separating them needs the window's nominal length, which never enters the scorer.

The ratio form is percentUsed / (elapsed / windowLength) — 1.0 means "exactly on pace to finish the window", above 1.0 means it runs out before it refills. One caveat from implementing this elsewhere: elapsed has to be floored (~15% of the window) or a freshly-reset window projects its first burst into a fake emergency.

Worth noting src/lib/quota/burnRate.ts already computes exactly this — computeBurnRateFromWindow — but it serves the quota-pool subsystem (redisQuotaStore / sqliteQuotaStore) and isn't reachable from buildAutoCandidates. So this may be less "add a burn-rate estimator" and more "route the existing one into scoring."

2. An unreadable quota scores as full

quotaRemainingPercentFromQuota (combo/comboPredicates.ts:432):

if (!quota || typeof quota !== "object") return 100;   // :433
...
return 100;                                             // :452

A quota object that is missing, malformed, or carries no recognisable percentUsed yields quotaRemaining = 100 → quota: 1.0, the best attainable score on that axis. The failure mode is that a provider whose usage endpoint is having a bad day doesn't merely stay eligible — it outranks every provider that honestly reported 60% consumed.

The asymmetry I'd suggest is that a telemetry failure is evidence about the telemetry, not about the provider. So it probably shouldn't do either extreme:

• it shouldn't rewrite to the most attractive value (today's behaviour), and
• it shouldn't evict traffic already running there either, since that pays a prompt-cache reset to avoid an outage that may not exist.

"Rank below any real reading, but don't move what's already there" splits those. Hard exhaustion and a failed health probe are different — those are evidence about the provider and should still move traffic.

#12317's PR body makes precisely this argument for reliability ("a corrupt reading means nothing observed, not fails every call"). This is the same principle applied to the quota axis.

3. A partial reading is a hole, not a smaller plan

Same function, :440-447:

for (const windowInfo of Object.values(windows)) {
  if (!windowInfo || typeof windowInfo !== "object") continue;
  const percentUsed = Number(windowInfo.percentUsed);
  if (!Number.isFinite(percentUsed)) continue;          // :443
  ...
  minRemaining = Math.min(minRemaining, remaining);
}

A provider that normally reports session + weekly + monthly and this time reports only session gets scored on the session window alone. Since the aggregate is min() over the windows that did parse, a missing window can only make the provider look better — best case wins by omission.

What makes this feel like an oversight rather than a design choice is that the codebase already has the concept: #10095 added fractionReported to QuotaInfo in src/domain/quotaCache.ts precisely so a window upstream didn't report can't be read as a real value, and isExhausted honours it. comboPredicates.ts doesn't reference fractionReported at all. The scoring path seems to have missed the fix the exhaustion path got.

Suggestion: track each provider's expected window set, and mark a reading incomplete when one is absent — incomplete being distinct from both "healthy" and "exhausted".

4. Failure kind carries routing information that failure rate discards

#12317 added reliability = 1 - failureRate as a single blended rate. That's a clear improvement over errorRate being declared and read nowhere. But the kinds fold together, and they predict differently:

• a 429 says the provider is out of quota — persistent, often for hours, and highly predictive of the next request
• a 5xx says something transient happened — weakly predictive
• a failed usage-endpoint read says nothing about the provider at all

My instance is the argument for splitting them. Over 30 days, 74,688 of 839,817 attempts were upstream_429 — nine times the next failure class. Concentrated: opencode-zen 67.8% of its attempts, gemini 58.5%, openrouter 45.3%, against ollama-cloud at 0.2%. Those rates persist for days, not minutes.

Correlating call_logs by correlation_id over four days (41,671 requests): 15.1% hit at least one 429. The fallback chain rescued 93% of them, so the reliability cost is small — but those 15% of requests consumed 47% of all upstream attempts (5.49 attempts each vs 1.12 for clean requests), and burned ~84 min/day of wall-clock in rejected round-trips.

That's the shape of the cost: not failures, waste. And a blended rate can't express "this provider will reject you specifically because it's out of quota, and will keep doing so until its window resets" — which is the one failure kind whose duration is knowable in advance.

---

Where this might go

If any of these land, my instinct is the order is (2), (3), (1), (4) — ascending in blast radius. (2) and (3) are small and local to comboPredicates.ts; (3) arguably just extends #10095's fractionReported into the scoring path. (1) needs window length plumbed to the scorer or burnRate.ts routed in. (4) is a design conversation and may belong under the failure-aware re-ranking already sketched in #12384 rather than as its own change.

Happy to be told any of these are deliberate trade-offs I've misread — I'd rather learn the reasoning than argue for a change. And if any are worth pursuing, I'm glad to open focused issues rather than keeping it in one thread.

[diegosouzapw]

@shannonlowder this is a careful read and I checked all four against release/v3.8.51, which is where a fix would land. Every one holds; only the line numbers moved.

1. Window length never reaches the score. Confirmed. calculateScore is a linear sum (open-sse/services/autoCombo/scoring.ts:160-185), quota enters as quotaRemaining / 100 (:323) and resetWindowAffinity as a separate term. computeBurnRateFromWindow (src/lib/quota/burnRate.ts:41) exists and is only referenced by the quota stores; nothing under open-sse/services/combo/ or autoCombo/ imports it. So yes: route the existing estimator, do not write a second one. Your floor on elapsed is the right caveat.
2. Unreadable quota scores as full. Confirmed. quotaRemainingPercentFromQuota returns 100 for a missing/malformed object and again at the end when nothing parsed (open-sse/services/combo/comboPredicates.ts:478 and :491). Your asymmetry ("rank below any real reading, do not evict") matches the reasoning feat(auto-combo): declare observed reliability as a scoring factor #12317 already applied to reliability.
3. Partial reading is a hole. Confirmed. The per-window loop now lives in open-sse/services/antigravityQuotaFamily.ts:80-98: unparseable windows are skipped, the aggregate is min() over what parsed, and fractionReported is not referenced anywhere in the scoring path (rg fractionReported open-sse/services/ hits only genericQuotaFetcher.ts and usage/antigravity.ts). The exhaustion path honours it; the scorer does not.
4. Failure kind vs rate. Confirmed as designed: reliabilityFactor is 1 - boundedRate(failureRate ?? errorRate) (scoring.ts:297-301), one blended number. Your 429 concentration data (67.8% on opencode-zen, 58.5% on gemini) is the strongest argument I have seen for splitting it, and the "47% of upstream attempts for 15% of requests" cost is the number that should headline the issue.

Your ordering is the one I would use too: (2) and (3) are local to two functions and can ship with unit tests against the exact inputs above; (1) needs the window length plumbed into the candidate; (4) is a design change and belongs in the #12384 discussion rather than as a standalone fix.

Please open (2) and (3) as separate issues, and (1) as a third; you have the instance data and the line references, and the issue should carry your name. Tag them enhancement and mention this thread. If you would rather I open them, say so and I will, with this discussion as the source.

[shannonlowder]

Thanks for checking this! I'll get on it! Let me know if you want my developer agent to try to pick up the work and make a first pass.

[diegosouzapw]

Yes, please do, @shannonlowder. One PR per issue, targeting release/v3.8.51, each carrying a unit test that fails on the current tip with the exact inputs from your write-up (the missing-window and malformed-quota cases are two-line fixtures). Whoever or whatever writes the first pass is your call; what gets reviewed is the diff and the test, and it is your name on it either way. If it is agent-written, read it before opening the PR, since the reviewer will.

[shannonlowder]

Opened as agreed: #15347 (unreadable quota scores as full), #15348 (partial reading is a hole; builds on #15347), #15349 (window length / route in burnRate.ts). One PR per issue against release/v3.8.52 (the current highest release branch), starting with #15347. #15347 includes one extra input found while re-verifying: percentUsed: null coerces to 0 via Number(null), so it also reads as full. Please add the enhancement label when you get a chance.

[diegosouzapw]

Thanks @shannonlowder, I see #15347, #15348 and #15349, and the extra input you found on #15347 (percentUsed: null coercing to 0 through Number(null), so it also reads as full) is a good catch. On the labels: #15348 and #15349 already carry enhancement. #15347 is labeled bug, which fits it better than enhancement, since it's a defect in how an unreadable quota is scored, so I've left it as is. One PR per issue against release/v3.8.52, starting with #15347, is the right order.