Skip to content

feat(auto-routing): resolve model aliases before benchmarking - #7582

Merged
iscekic merged 2 commits into
mainfrom
feat/auto-routing-bench-alias
Oct 15, 2026
Merged

iscekic merged 2 commits into
mainfrom
feat/auto-routing-bench-alias

Conversation

@iscekic

@iscekic iscekic commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

  • Publish optional alias_target slugs for ~ aliases in the model catalog.
  • Resolve platform and owner-pool selections before registry admission, take reasoning variants from the concrete target, and deduplicate exact resolved model/variant pairs. Missing targets keep the alias id. Retargeted aliases request the new concrete measurement.
  • Share owner and platform measurements while mapping registration/status responses back to each requested pool model. Retry requests resolve through the same catalog.
  • Keep routing tables, prices, and custom candidate capability lookups concrete. The benchmark custom-table assembler already restricts candidates to requested ready/current exact pairs and provenance, so the serving worker no longer drops concrete candidates by comparing them against saved aliases.
  • Document selection, deduplication, and rollout behavior.

Rollout

Deploy the auto-routing serving worker first, then the auto-routing-benchmark worker. Older serving workers filter concrete custom-table candidates against saved alias ids. The new serving worker supports both existing alias-id tables and new concrete-id tables; both shapes have regression coverage. Publish the gateway catalog field before expecting alias resolution to take effect.

Verification

  • Focused benchmark Vitest files: platform-models, admin, orphan-claim-sql, profile-runs, custom-routing-table — 128 tests passed, including real SQLite retarget reconciliation and concrete-id publication/pricing.
  • Serving src/index.test.ts — 104 tests passed, including old/new benchmark-worker custom-table compatibility.
  • Shared OpenRouter provider index.test.ts — 34 Jest tests passed using the web configuration with database worker setup omitted for this unit-only run. Includes upstream object/absent alias metadata, database overwrite, and omission of upstream-only alias targets. The local test database was not running; existing optional benchmark-summary lookups emitted connection-refused warnings and used their normal fallback.
  • pnpm --filter auto-routing-benchmark typecheck, pnpm --filter auto-routing typecheck, and pnpm --filter @kilocode/web-shared typecheck — passed.
  • Type-aware oxlint on touched benchmark, serving, catalog-provider, and organization directories — zero warnings/errors.
  • pnpm format:changed and staged whitespace check — passed.

No production benchmark was started. CI and review are not awaited.

@iscekic iscekic self-assigned this Oct 15, 2026
Comment thread services/auto-routing/src/decide.ts Resolved
@kilo-code-bot

kilo-code-bot Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The alias-sanitization fix resolves both prior catalog-parsing defects; one advisory serving-worker allow-list concern remains in the unchanged decide.ts (deliberately kept per the PR design).

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
services/auto-routing/src/decide.ts 86 The serving worker no longer independently enforces pool membership; custom-table candidates are trusted from the benchmark origin alone (carried forward from the prior review, declined by design).
Files Reviewed (3 files)
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/index.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/index.ts
  • packages/web-shared/src/lib/organizations/organization-types.ts

Fix these issues in Kilo Cloud

Previous Review Summary (commit 18321a8)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 18321a8)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

A new alias_target: z.string() field on the shared OpenRouterModelSchema conflicts with the raw OpenRouter payload (an object), breaking upstream catalog parsing.

Overview

Severity Count
CRITICAL 1
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

CRITICAL

File Line Issue
packages/web-shared/src/lib/organizations/organization-types.ts 352 alias_target: z.string() is also applied to raw upstream data (getRawOpenRouterModels), where OpenRouter returns an object ({ name, slug }). The array parse now fails, emitting error-level Sentry events and bypassing model validation.

WARNING

File Line Issue
packages/web-shared/src/lib/ai-gateway/providers/openrouter/index.ts 168 ...model carries the raw upstream alias_target object; when DB metadata lacks the field, the object leaks into the served catalog and breaks clients/benchmark parsing.

SUGGESTION

File Line Issue
services/auto-routing/src/decide.ts 86 Removing the pool-membership filter drops the worker's independent allow-list check; candidates are now trusted from the benchmark origin only.
Files Reviewed (12 files)
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/index.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/index.ts - 1 issue
  • packages/web-shared/src/lib/organizations/organization-types.ts - 1 issue
  • services/auto-routing-benchmark/README.md
  • services/auto-routing-benchmark/src/admin.test.ts
  • services/auto-routing-benchmark/src/admin.ts
  • services/auto-routing-benchmark/src/orphan-claim-sql.test.ts
  • services/auto-routing-benchmark/src/platform-models.test.ts
  • services/auto-routing-benchmark/src/platform-models.ts
  • services/auto-routing-benchmark/src/profile-runs.test.ts
  • services/auto-routing/src/decide.ts - 1 issue
  • services/auto-routing/src/index.test.ts

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash · Input: 67.7K · Output: 16.3K · Cached: 956.9K

Review guidance: REVIEW.md from base branch main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants