Repository navigation
refactor(auto-routing-benchmark): mint benchmark tokens as an internal bot - #8092
Merged
Merged
Conversation
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Contributor
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Executive SummaryThe port onto main's Overview
Issue Details (click to expand)WARNING
Files Reviewed (10 files)
Fix these issues in Kilo Cloud Previous Review Summaries (3 snapshots, latest commit 44bf33c)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 44bf33c)Status: 3 Issues Found | Recommendation: Address before merge Executive SummaryThe merge of Overview
Issue Details (click to expand)CRITICAL
WARNING
Files Reviewed (19 files)
Fix these issues in Kilo Cloud Previous review (commit 02468fa)Status: 3 Issues Found | Recommendation: Address before merge Executive SummaryThe incremental fix correctly reports a non-bot identity override as HTTP 400 and makes the missing-org check lazy; the remaining blocker is that pre-deploy run rows still snapshot a non-bot user, plus two stale lines in the token policy doc. Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (7 files)
Fix these issues in Kilo Cloud Previous review (commit 4c42072)Status: 6 Issues Found | Recommendation: Address before merge Executive SummaryThe refactor is internally consistent, but in-flight decider runs snapshot a non-bot user id that the new mint rejects, so they will dead-letter after the deploy resumes the queues; admin error status and two stale policy-doc lines are also worth fixing. Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (18 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 137.2K · Output: 33.1K · Cached: 4.8M Review guidance: REVIEW.md from base branch |
eshurakov
approved these changes
Oct 19, 2026
…factor/benchmark-internal-bot-token
…ternal-bot-token # Conflicts: # ENVIRONMENT.md # apps/web/src/routers/admin/internal-bot-users-router.ts # docs/token-issuance-policy.md # packages/web-shared/src/lib/config.server.resource-tokens.test.ts # packages/web-shared/src/lib/config.server.ts
…in's benchmark-token helper Main moved fetchBenchmarkUserToken into benchmark-token.ts and added the executable model catalog as a caller, so a merge left run.ts and admin.ts with stale code. Rebuild the PR on main's structure: every mint goes to /api/internal/bot-token with BOT_TOKEN_MINT_SECRET and a required org, the catalog always uses the org scope, and a legacy run row without an org fails at initialization instead of minting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The decider benchmark now runs as the internal bot user
bot-internal-benchmark. It mints that user's token fromPOST /api/internal/bot-token, added in #8088. It no longer mints a token for a personal user, so it stops sharing that user's per-user gateway limit.fetchBenchmarkUserToken(services/auto-routing-benchmark/src/benchmark-token.ts) calls${KILO_WEB_API_BASE_URL}/api/internal/bot-token. It authenticates with a new Secrets Store binding,BOT_TOKEN_MINT_SECRET(secret nameBOT_TOKEN_MINT_SECRET, in the same store as the other bindings), and sends{ userId, organizationId }.organizationIdis now a required parameter.run.ts,/admin/debug-cli, and the executable model catalog (executable-models.ts), which the queue drain, run initialization, subscriptions and profile refresh use. The catalog always reads the org catalog, because the identity always has an org.INTERNAL_API_SECRET_PRODstays where other routes still need it: admin auth,production-profile, and grader usage.worker-configuration.d.tswas regenerated withpnpm types.DEFAULT_BENCHMARK_USER_IDis nowbot-internal-benchmark. The default org is unchanged.benchmarkUserIdoverride that is not abot-internal-*user now makesresolveBenchmarkIdentitythrow aBenchmarkIdentityError. It does not silently fall back to the default. Starting a run, a queue drain and/admin/debug-clireturn a 400 before anything is claimed. The MCP catalog mint, which shares the same identity, returns the error as a 502.resolveBenchmarkIdentityalways supplies an org. A decider job whose run row has no org snapshot, such as an old row, now refuses to mint a token. Before, it sent the request without an org. A chunk whose cases are all recorded still finishes, because it needs no mint. An uninitialized run row without an org fails at initialization, the same way as a row without a user.apps/web/src/app/api/internal/auto-routing-benchmark/token/(route and test). With it went the now-unusedbenchmarkresource-token family (BENCHMARK_RESOURCE_TOKENS_ENABLED) and its doc and test entries. Updated the README and the token-issuance policy map.Verification
services/auto-routing-benchmark: typecheck, oxlint and the full vitest suite pass. The new tests check the mint request (URL, dedicated secret,{ userId, organizationId }body), that no token is minted for a run without an org, the 400 from/admin/debug-cli, and that run initialization fails a row without an org.packages/auto-routing-contracts: typecheck and vitest pass, including a test that non-bot overrides are rejected (a human id, a per-org product bot, a lookalike prefix).services/auto-routing: typecheck passes.apps/web: typecheck passes. Jest passes forsrc/app/api/internal/**(including the MCP catalog mint and its new non-bot test) andsrc/lib/bot-users.Deploy order
bot-internal-benchmarkin the benchmark org from the admin Internal Bot Users page (from feat(admin): internal bot users with short-lived token minting #8088). Give it credits.BOT_TOKEN_MINT_SECRETwith one value in two places: the web environment and Secrets Store secretBOT_TOKEN_MINT_SECRET. LeaveINTERNAL_BOT_RESOURCE_TOKENS_ENABLEDunset, which keeps the legacy token shape. Before you set it, test a modern minted token against/api/profileand the organization models endpoint.benchmarkUserIdoverride in the benchmark config, or set it tobot-internal-benchmark. A non-bot override blocks new runs and the MCP catalog mint once this ships.runningdecider run finish, or fail it from the admin panel. A run row written before this change holds the old non-bot user, and the new mint refuses that user. Do not merge while such a run still has chunks queued.auto-routing-benchmarkworker deploy to finish.