[iscekic]

Summary

Web app slice of ADR 0005 (custom auto routers). This PR adds:

• the owner-facing REST API for custom routers,
• the web editor,
• the models-catalog annotation.

It is gated by the PostHog flag kilo-auto-custom-routers.

REST API (/api/auto-routing/custom-routers)

• Paths and payloads come from the shared contracts (AUTO_ROUTING_CUSTOM_ROUTERS_PATH, customRouterPath, customRouterClientEntriesPath, and the request and response schemas).
• Endpoints:
  • GET lists routers. POST creates one.
  • GET, PATCH, and DELETE work on /{scope}/{slug}.
  • POST and DELETE on /{scope}/{slug}/client-entries register and remove client entries. The CLI uses these.
• Errors use CustomRouterErrorResponseSchema, and every response status comes from the contract HTTP status table.
• Authentication: getUserFromAuth accepts both a web session and a Kilo user bearer token.
  • With a bearer token, the organization context comes from the token or X-KiloCode-OrganizationId.
  • The web UI passes ?organizationId= instead.
  • Organization access is checked with ensureOrganizationAccess.
• Ownership always comes from the authenticated identity. Owner fields in the request body are ignored.
• Editing an organization router needs one of the roles in ORGANIZATION_BILLING_ROLES (owner, admin, billing manager). It also needs an active subscription or trial. Other members get read-only summaries with canEdit: false.
• Limits:
  • The contract schemas enforce the slug pattern, the reserved org slug, 1–20 unique entries, and that the default entry is in the pool.
  • Writes take a per-owner advisory lock. Under that lock, the API checks the 10-router cap and slug uniqueness. The unique indexes stay as a backstop.
  • New gateway entries are checked against the owner's catalog: the organization policy applies, and Kilo-stored direct-BYOK IDs are allowed.
• Platform admission: when a router's entries or base router change, the platform-servable gateway entries are registered with the benchmark worker (/admin/profiles/register), in batches of 10, with router = baseRouter.
  • If the owner hits the benchmark quota, nothing is written. The response is quota_exceeded (HTTP 429), and the error object carries retryAt.
• Entry status: the status for each router comes from the benchmark worker (CUSTOM_ROUTER_STATUS_PATH with CustomRouterBenchmarkContextSchema).
  • clientEvalEntryKeys lists every client entry and every direct-BYOK gateway entry. This is the same rule the serving worker uses.
  • needsClientEval is true when the CLI must evaluate the entry and not every variant is ready under the current engine identity.
  • If the status call fails, the list still returns, with empty variants for each entry.
  • Removing a client entry is idempotent. Removing the default entry is rejected; choose another default first.

UI

• New CustomRoutersCard with a create/edit dialog.
  • The dialog edits the name, slug (shown with a preview of the model ID), base router, mode, pool, and default model.
  • Each pool entry shows its status, with a client-eval notice that includes the kilo auto eval command.
  • Organization members who cannot edit see a read-only view.
• Where it appears:
  • On the profile page, it is gated server-side on the flag.
  • In the organization's Providers & Models › Auto routing tab, it hides itself when the API answers 404.
• AutoRoutingModeCard is unchanged.

Catalog

• /api/openrouter/models now lists the flagged user's routers:
  • their personal routers as kilo-auto/custom/<slug>,
  • in an organization context, also the organization's routers as kilo-auto/custom/org/<slug>.
• Each router entry is built with customRouterAutoModel from the base router's auto model. Its autoRouting.models lists the gateway entries visible in the caller's catalog.
• The flag lookup runs only when the owner has routers.

Flag

• kilo-auto-custom-routers is evaluated per user, with a strict boolean check (isReleaseToggleEnabled); it is always on in development.
• A user without the flag gets router_not_found (404) on every route. Their routers do not appear in the catalog.

Dependencies

• The benchmark worker's CUSTOM_ROUTER_STATUS_PATH route and frontier admission come in the benchmark worker PR of this stack. Until that deploys, the list returns empty variants for each entry, and creating a router fails when the worker rejects admission.
• Gateway resolution of kilo-auto/custom/* and removal of the old pool settings come in later stacked PRs.

Verification

On a work VM:

• pnpm typecheck passes in packages/web-shared and apps/web.
• pnpm lint passes in apps/web.
• These Jest suites pass:
  • src/app/api/auto-routing/**, including the new custom router route tests, which run against the real database,
  • the profile page test,
  • src/tests/openrouter-models.test.ts,
  • the web-shared auto-routing-models, handlers/models, handlers/organization-models, auto-routing-benchmark-admin-client, and direct-byok tests.
• oxfmt reports no changes on the changed files.

[kilo-code-bot]

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

Commit a65e5730 closes the last retry gap by also retrying a worker 5xx within the shared admission deadline; the only remaining item is the carried-forward deployment-ordering gate for catalog-advertised kilo-auto/custom/* IDs.

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts	99	Carried forward and re-verified at a65e5730 (file unchanged since e870f13f): addCustomRouterModels lists kilo-auto/custom/<slug> IDs, but isKiloAutoModel (auto-model/index.ts:206) still matches only AUTO_MODELS and ORG_AUTO_MODEL, and llm-proxy.ts:390 routes through it, so a request selecting a custom ID is not resolved as an auto router. The author states gateway resolution ships in #7809 within the same deploy window, so this is a deployment-ordering gate, not a defect in this commit. Already covered by an active inline comment; reported here only.

Verification Notes

• Incremental diff c70847f9..HEAD is one commit (a65e573038) touching three files: the 5xx retry in requestPlatformBenchmarks, its tests, and the ADR sentence.
• Previous SUGGESTION (retry covered only rejected fetch calls) is resolved: the fulfilled branch now retries once when !first.ok && first.status >= 500, guarded by !signal.aborted; quota (429) and 4xx results are still not retried, matching the contract result union in auto-routing-benchmark-admin-client.ts:147.
• Verified the retry reuses the single AbortSignal.timeout signal, so the ten-second deadline still bounds total admission time, and the retry rejection is routed through the existing captureException / internal_error path.
• Tests are consistent with the implementation: the default mock resolves { ok: true }, so the new test.each 503 case succeeds on the second call, the rejected-request case still works, and the persistent-failure tests now assert exactly two registration attempts.
• ADR 0005 wording now matches the code ("fails in transit or gets a worker 5xx before the deadline is retried once").
• No image references were added to markdown files in this revision, and no memory-lifetime changes (subscriptions, listeners, caches, timers) are introduced.

Files Reviewed (3 files)

• apps/web/src/lib/auto-routing-custom-routers.ts
• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• docs/adr/0005-custom-auto-routers.md

Fix these issues in Kilo Cloud

Previous Review Summaries (6 snapshots, latest commit c70847f)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit c70847f)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

Commit c70847f9 adds a bounded one-shot retry, a ten-second admission deadline, and Sentry capture to benchmark registration, resolving the previous slow-admission finding; one gap remains in that retry's scope, plus a carried-forward deployment-ordering gate for catalog-advertised kilo-auto/custom/* IDs.

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	1

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts	99	Carried forward and re-verified at c70847f9 (file unchanged since e870f13f): addCustomRouterModels lists kilo-auto/custom/<slug> IDs, but isKiloAutoModel (auto-model/index.ts:206) still matches only AUTO_MODELS and ORG_AUTO_MODEL, and llm-proxy.ts:390 routes through it, so a request selecting a custom ID is not resolved as an auto router. The author states gateway resolution ships in #7809 within the same deploy window, so this is a deployment-ordering gate, not a defect in this commit. Already covered by an active inline comment; reported here only.

SUGGESTION

File	Line	Issue
apps/web/src/lib/auto-routing-custom-routers.ts	337	The new retry wraps only rejected fetch calls. registerOwnerBenchmarkProfiles resolves for every HTTP error status ({ ok: false, status, error }), so a transient worker 502/503/504 still becomes internal_error with no retry, unlike the ADR's "fails in transit" intent.

Verification Notes

• Incremental diff e870f13f..HEAD is one commit (c70847f974) touching three files: the timeout constant (5_000 -> 10_000), a bounded retry guarded by signal.aborted, captureException on final failure, matching tests, and the ADR paragraph.
• Previous SUGGESTION at line 328 (opaque 500, no retry, no Sentry) is resolved: admission retries once while the shared deadline is live, uses a ten-second deadline, and reports the final failure to Sentry; tests cover a successful retry, a double failure, and a deadline abort with no retry.
• Verified the retry reuses the single AbortSignal.timeout signal, so the deadline still bounds total admission time and the abort guard prevents retrying after expiry.
• route.test.ts changes are consistent with the implementation: the new test.each message is a substring of an existing UNSUPPORTED_BASE_ROUTER_ERRORS fragment, toHaveBeenCalledTimes(1) holds under the abort guard, and the 11-entry test still expects two batches on one shared signal.
• No image references were added to markdown files in this revision.

Files Reviewed (3 files)

• apps/web/src/lib/auto-routing-custom-routers.ts - 1 issue
• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• docs/adr/0005-custom-auto-routers.md

Fix these issues in Kilo Cloud

Previous review (commit e870f13)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The new commit e870f13f intentionally serializes custom-router validation, benchmark admission, and the write under the per-owner advisory lock with a shared five-second admission deadline; that serialization fix is correct and covered by new concurrency tests, but a single slow admission now turns a save into an opaque 500 that may still spend quota, and the catalog still advertises kilo-auto/custom/* IDs this branch's gateway cannot resolve.

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	1

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts	99	Carried forward and re-verified at e870f13f: addCustomRouterModels lists kilo-auto/custom/<slug> IDs, but isKiloAutoModel (auto-model/index.ts:206) still matches only AUTO_MODELS and ORG_AUTO_MODEL, so a request selecting one falls through to the provider path with the virtual ID. The author states gateway resolution ships in #7809 within the same deploy window, so this is a deployment-ordering gate, not a defect in the new commit. Already covered by an active inline comment; reported here only.

SUGGESTION

File	Line	Issue
apps/web/src/lib/auto-routing-custom-routers.ts	328	The shared five-second admission deadline is converted by the .catch on line 338 into internal_error with no retry, no timeout-specific message, and no Sentry capture. One slow worker response rolls back the write while the worker may already have admitted the profiles and charged quota.

Verification Notes

• Incremental diff 26c9e696..e870f13f touches five files. The substantive change moves catalog I/O (gatewayEntryRejections) out of the lock and runs validateNewGatewayEntries plus requestPlatformBenchmarks inside the locked transaction for create and update; delete now also takes the owner lock, and admission gained a shared AbortSignal.timeout(5_000) passed through to registerOwnerBenchmarkProfiles.
• Previous finding "benchmark profiles are admitted before the owner lock" is resolved: admission now runs under lockOwner against the locked, merged definition (create at line 485, update at line 559), and the new concurrent admission tests plus rolls back timed-out admission and releases the owner lock cover the duplicate-slug, cap, stale-default, and locked-base-router races.
• The earlier "outbound HTTP runs inside the advisory-lock transaction" comment is now re-introduced by design, but bounded: the deadline covers all batches including body reads, and the residual quota-orphan risk is documented in ADR 0005. Treated as an accepted tradeoff rather than re-raised.
• The gatewayEntryRejections / validateNewGatewayEntries split preserves prior validation semantics (entries whose key is already stored are still skipped via existingKeys), and entries.<index> now reports the real index in the merged pool, fixing the previous index bug.
• Only one production caller of registerOwnerBenchmarkProfiles exists, and it passes an explicit signal, so the new default parameter changes no other caller. No image references were added to markdown files in this revision.

Files Reviewed (5 files)

• apps/web/src/lib/auto-routing-custom-routers.ts - 1 issue
• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.test.ts
• docs/adr/0005-custom-auto-routers.md

Fix these issues in Kilo Cloud

Previous review (commit 26c9e69)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The incremental diff for this revision is empty — all 20 changed PR files are content-identical to the previously reviewed commit 3de2f0a — so no new defects were raised; the two still-open findings are that flagged users can select kilo-auto/custom/* model IDs the gateway cannot yet resolve, and benchmark admission runs outside the owner lock, which can spend quota for a router that is never written.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts	99	The catalog now advertises kilo-auto/custom/… IDs that the gateway does not resolve in this revision; isKiloAutoModel does not recognize them, so a request selecting one falls through to the provider path with the virtual ID. Resolution is intentionally deferred to a later stacked PR, so deployment with the gateway-resolution change (or keeping these IDs out of the catalog) is required before the flag is enabled.
apps/web/src/lib/auto-routing-custom-routers.ts	466	Benchmark profiles are admitted before the per-owner advisory lock and the DB write. A concurrent same-owner create/update that wins the in-transaction slug/cap re-check, or a locked re-merge that differs from the pre-transaction plan, leaves the worker profiles admitted for a router state that is never saved, spending the owner's benchmark quota.

Verification Notes

• git diff 3de2f0a..HEAD produces no changes for any file in gh pr diff 7790 --name-only (the branch was rebased; the same content is present at both commits), so this pass has no new changed lines to review.
• Re-verified against current HEAD (26c9e696): quota rejection now uses the contract quota_exceeded code with error.retryAt and every status derives from CUSTOM_ROUTER_ERROR_HTTP_STATUS (auto-routing-custom-routers.ts:326-330); the card guards on query.isError && query.data === undefined and shows a refresh alert; the poll filters routers by scope; usableCustomRouters degrades to [] and reports to Sentry. Those findings are resolved.
• Both open findings above already carry active inline comments, so they are reported here only and not re-posted.

Files Reviewed (20 files)

• apps/web/src/app/(app)/profile/page.test.ts
• apps/web/src/app/(app)/profile/page.tsx
• apps/web/src/app/api/auto-routing/custom-routers/[scope]/[slug]/client-entries/route.ts
• apps/web/src/app/api/auto-routing/custom-routers/[scope]/[slug]/route.ts
• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• apps/web/src/app/api/auto-routing/custom-routers/route.ts
• apps/web/src/components/auto-routing/CustomRouterDialog.tsx
• apps/web/src/components/auto-routing/CustomRoutersCard.tsx
• apps/web/src/components/auto-routing/custom-router-display.ts
• apps/web/src/components/organizations/providers-and-models/OrganizationProvidersAndModelsPage.tsx
• apps/web/src/lib/auto-routing-custom-routers.ts
• apps/web/src/tests/openrouter-models.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-model/index.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-models.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts
• packages/web-shared/src/lib/ai-gateway/custom-routers.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models.test.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models.ts

Fix these issues in Kilo Cloud

Previous review (commit 3de2f0a)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental changes (contract quota_exceeded code plus error.retryAt, the web CustomRouterApiError/response refactor, and the unsupported-base-router mapping) are consistent across the contracts, API, and tests, and every response status now derives from CUSTOM_ROUTER_ERROR_HTTP_STATUS; no new defects found.

Files Reviewed (5 files)

• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• apps/web/src/lib/auto-routing-custom-routers.ts
• packages/auto-routing-contracts/src/custom-router.test.ts
• packages/auto-routing-contracts/src/custom-router.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.test.ts

Verification Notes

• Previous findings re-verified against current HEAD (3de2f0a): benchmark registration / catalog validation now run before the advisory-lock transaction; the card guards on query.isError && query.data === undefined; the poll filters routers by scope; usableCustomRouters degrades to [] on failure; quota rejection uses the new quota_exceeded (HTTP 429) code with error.retryAt. All resolved.
• The refactor removing the status/extra override preserves behavior: forbidden→403, invalid_request→400, unauthorized→401, quota_exceeded→429, and no call site relies on the removed fields.
• The retryAt field moved from the response top level to error.retryAt; the only in-repo consumers (custom-router-display.ts) read error.message, and no other caller reads a top-level retryAt for this endpoint.

Previous review (commit 5cf9a43)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
apps/web/src/lib/auto-routing-custom-routers.ts	326	Quota admission rejection still returns HTTP 429 with the invalid_request code, while CUSTOM_ROUTER_ERROR_HTTP_STATUS maps invalid_request to 400 and the PR states errors use that contract table. Author acknowledged this in-thread as an intentional, documented exception (only the web UI reaches this path; the contracts package has no quota code). Non-blocking.

Files Reviewed (5 files)

• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• apps/web/src/components/auto-routing/CustomRoutersCard.tsx
• apps/web/src/lib/auto-routing-custom-routers.ts - 1 issue
• packages/web-shared/src/lib/ai-gateway/handlers/models.test.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models.ts

Fix these issues in Kilo Cloud

Previous review (commit 182a80c)

Status: 5 Issues Found | Recommendation: Address before merge

Executive Summary

The server correctly derives ownership, enforces org access and subscription checks, and scopes every query by owner; the main risks are a per-owner advisory-lock transaction that spans outbound HTTP, a UI error guard that can discard already-loaded data, and an optional catalog annotation that can fail the whole models endpoint.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	3

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/lib/auto-routing-custom-routers.ts	458	Benchmark registration (and, on update, catalog validation) runs inside the transaction holding the per-owner pg_advisory_xact_lock; the benchmark fetch has no timeout, so a hung worker serializes that owner's writes and pins a pooled connection. Create validates outside the lock while update validates inside it.
apps/web/src/components/auto-routing/CustomRoutersCard.tsx	111	if (query.isError) without !query.data (unlike the sibling AutoRoutingModeCard). The card's 15s poll and post-save invalidation can fail a background refetch, which then replaces the loaded router list with the destructive error card.

SUGGESTION

File	Line	Issue
apps/web/src/components/auto-routing/CustomRoutersCard.tsx	52	hasPendingEntries receives the unfiltered router list while rendering is filtered by scope, so a personal router stuck pending keeps the organization card polling every 15s indefinitely.
packages/web-shared/src/lib/ai-gateway/handlers/models.ts	65	listUsableCustomRouters shares the handler's single try/catch, so a failure loading routers (DB error, or rowOwner throwing on an ownerless row) fails the entire /api/openrouter/models response with 500 instead of degrading to [].
apps/web/src/lib/auto-routing-custom-routers.ts	324	Quota rejection returns HTTP 429 with the invalid_request code, which the contract table maps to 400; a CLI branching on the error code would treat it as a validation error.

Files Reviewed (16 files)

• apps/web/src/app/(app)/profile/page.test.ts
• apps/web/src/app/(app)/profile/page.tsx
• apps/web/src/app/api/auto-routing/custom-routers/route.ts - 1 issue (in shared lib)
• apps/web/src/app/api/auto-routing/custom-routers/route.test.ts
• apps/web/src/app/api/auto-routing/custom-routers/[scope]/[slug]/route.ts
• apps/web/src/app/api/auto-routing/custom-routers/[scope]/[slug]/client-entries/route.ts
• apps/web/src/components/auto-routing/CustomRouterDialog.tsx
• apps/web/src/components/auto-routing/CustomRoutersCard.tsx - 2 issues
• apps/web/src/components/auto-routing/custom-router-display.ts
• apps/web/src/components/organizations/providers-and-models/OrganizationProvidersAndModelsPage.tsx
• apps/web/src/lib/auto-routing-custom-routers.ts - 2 issues
• apps/web/src/tests/openrouter-models.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-model/index.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-benchmark-admin-client.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-models.ts
• packages/web-shared/src/lib/ai-gateway/custom-routers.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models.ts - 1 issue

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash · Input: 44K · Output: 5.8K · Cached: 350.8K}

_{Review guidance: REVIEW.md from base branch main}