Repository navigation
[AISAGE-857] fix(byok): enforce the organization provider allow-list on Vercel provider keys - #9215
Conversation
…ise-provider-policy # Conflicts: # packages/web-shared/src/lib/ai-gateway/byok/index.ts
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Executive SummaryBoth prior findings are resolved in current code: Assumptions: Files Reviewed (4 files)
Previous Review Summaries (2 snapshots, latest commit 5887ccd)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 5887ccd)Status: No Issues Found | Recommendation: Merge Executive SummaryThe incremental change resolves the remaining finding: Assumptions: Files Reviewed (2 files)
Also re-checked for the carried-forward findings: Previous review (commit 45684dc)Status: 3 Issues Found | Recommendation: Address before merge Executive SummaryThe new Vercel BYOK allow-list check is skipped when the model decision is allowed without Assumptions: Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (15 files)
Reviewed by deepseek-flash-latest · Input: 89.1K · Output: 19.9K · Cached: 1.7M Review guidance: REVIEW.md from base branch |
…ys in the balance check
…ise-provider-policy
chrarnoldus
left a comment
There was a problem hiding this comment.
Code Review Summary
Status: 2 Issues Found | Recommendation: Address before merge
Executive Summary
The new filtering can silently switch a request with an undecryptable eligible BYOK key to organization-credit billing, and can reject Organization Auto targets explicitly permitted by the default model-access policy.
Assumptions: none. Both findings were reproduced using actual revision sources with mocked database/catalog/KMS boundaries. Full local suites and live-provider tests were not rerun; CI is green at the reviewed head.
Overview
| Severity | Count |
|---|---|
| CRITICAL | 0 |
| WARNING | 2 |
| SUGGESTION | 0 |
Issue Details (click to expand)
WARNING
| File | Line | Issue |
|---|---|---|
packages/web-shared/src/lib/ai-gateway/providers/get-provider.ts |
424-426 | [P2] Post-decryption policy filtering can defeat the credential loader's fail-closed billing protection. |
packages/web-shared/src/lib/ai-gateway/byok/index.ts |
326-330 | [P2] Organization Auto rejects targets explicitly granted by the default model-access policy. |
Files Reviewed (17 files)
apps/web/src/components/organizations/byok/BYOKKeysManager.tsxapps/web/src/lib/cloud-agent-next/balance-check-eligibility.test.tsapps/web/src/lib/cloud-agent-next/balance-check-eligibility.tspackages/web-shared/src/lib/ai-gateway/byok/availability.test.tspackages/web-shared/src/lib/ai-gateway/byok/availability.tspackages/web-shared/src/lib/ai-gateway/byok/index.tspackages/web-shared/src/lib/ai-gateway/handlers/embeddings.test.tspackages/web-shared/src/lib/ai-gateway/handlers/embeddings.tspackages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.tspackages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.tspackages/web-shared/src/lib/ai-gateway/providers/get-provider.preview.test.tspackages/web-shared/src/lib/ai-gateway/providers/get-provider.tspackages/web-shared/src/lib/ai-gateway/providers/getEmbeddingProvider.test.tspackages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.test.tspackages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.tspackages/web-shared/src/lib/ai-gateway/providers/vercel/index.tspackages/web-shared/src/lib/organizations/group-policies/model-access/model-access.server.ts
Part of AISAGE-857: apply organization model policies to user BYOK.
Problem
In an organization context, a provider key routed through Vercel (for example Bedrock, Anthropic, Mistral, or Moonshot AI) bypassed the Enterprise provider allow-list.
applyVercelSettingspinned every organization key and dropped the policy'sonlylist.allowsInferenceProviderUserByokalso admittedorganization_providerdenials for those keys. The result was that a prompt could reach a provider the admin had not allowed. For example, with an org allow-list of["deepinfra"]and a Bedrock key,anthropic/claude-haiku-4.5was served by Bedrock.#9092 added this exemption on purpose ("the provider itself is considered trusted"). This PR changes that decision for Vercel provider keys. The provider allow-list is now enforced whenever it can name the key's provider, meaning the Vercel provider maps to an OpenRouter provider slug on the provider policy screen. Today that covers every Vercel BYOK provider. Direct BYOK keys, catalog mode included, are unchanged. No active production organization is affected: a production query for Enterprise organizations with an enabled Vercel provider key outside an org-wide or
selectedpolicy provider allow-list found none with recent activity. The trade-off: the #9092 case (allow-list["anthropic"]plus a Moonshot AI key for Kimi) is denied again unless the org addsmoonshotaito the allow-list.Fix
isVercelUserByokProviderAllowed(providerId, allowedProviders): a Vercel key is admitted when the policy doesn't restrict providers, when the policy can't name the key's provider, or when an allowed OpenRouter slug maps to it (amazon-bedrock→bedrock,google-vertex→vertex,google-ai-studio→google,z-ai→zai; acodestralkey counts asmistral).getProvidergets a newgetAllowedProvidersinput. Providers outside the allow-list are dropped before their keys are loaded or decrypted, so the request falls through to the next route: a gateway key, or Kilo's gateway with the allow-list applied. Because disallowed keys are never decrypted, an allowed key that fails to decrypt still fails the request closed instead of falling back to Kilo credits.llm-proxysupplies the member'seligibleProviderRoutes, or an empty set for anorganization_providerdenial. It also rejects withmodel_not_allowed, as defence in depth, if a Vercel route still carries a disallowed key.getEmbeddingProvider.hasInferenceProviderUserByokForModel,hasEnabledUserByokForModel) apply the same rule, so they no longer offer models that requests would reject.model_accesspolicy is evaluated with the settings being saved. A model that the default policy explicitly grants stays a valid target, matching request-time behavior. This replaces the Moonshot-key admission that the new rule removes.computeCloudAgentNextBalanceCheckEligibility) resolves the member decision for every organization request. A provider key that routing would drop no longer skips the $1 minimum.Manual: how policies affect models and providers
Private custom LLMs are out of scope. "Before" means
mainwithout this PR, which already includes #9092.1. Which policy applies
model_accesspolicies, combined as below.In an organization, routing uses only keys the organization stored. A member's personal keys are never used, so they can't get around the policy.
2. Who gets what in an Enterprise organization
Inputs:
model_deny_listandprovider_allow_list(the provider "ceiling").model_accesspolicy, plus the policy of every group the member belongs to. Each policy isall,none, orselected, whereselectedhasmodel_allow_listandprovider_allow_list.The member's grant:
all, and no policy isselected.selected. The member gets the union of every selectedmodel_allow_listandprovider_allow_list. The baseline is added on top when there is no default policy, the default isall, or any group policy isall.none, or the default isnoneand no group policy grants anything.Decision for one model, which yields allowed or denied plus a set of allowed provider routes. Provider routes are OpenRouter provider slugs: the model's providers in the current snapshot, intersected with the allowed providers.
neuralwatt/..., are exempt. They are allowed with no provider routes.organization_model). A model not in the snapshot is denied. Without a ceiling, the model is allowed with no route limit. With a ceiling, the routes are the model's providers that fall within the ceiling. An empty result is denied asorganization_provider.model_allow_listis allowed with no route limit. Neither the organization deny list nor the ceiling applies to it.provider_allow_list, it is allowed with those providers as routes. If the baseline is included, the baseline's routes are added. A baseline with no route limit lifts the limit.no_grant).organization_provideris the only denial reason that means "the model is allowed, but none of its providers are". A group's provider list never produces that reason: when a group's providers don't serve the model, the result isno_grant.3. How each route applies the decision
"Routes" below means the allowed provider routes from section 2.
getProviderorder)provider/modelnaming a direct provider)organization_providerdenial is admitted./responses)anthropic,bedrock,vertex,openai,mistral/codestral,moonshotai, ...)organization_providerdenial was admitted.amazon-bedrock→bedrock,google-vertex→vertex,google-ai-studio→google,z-ai→zai,mistral→mistral/codestral). Other keys are skipped and the next route takes the request. Anorganization_providerdenial leaves no allowed provider, so every such key is skipped. A provider the policy screen cannot name would stay exempt, but none exists today.only: routesonly: routesEmbeddings use the same decision. Before, a Vercel provider key was chosen before the decision was evaluated, and the routes were never applied to it. After, the routes filter keys in the same way, and a skipped key falls through to OpenRouter with
only: routes.The same rule is used wherever the product predicts routing. Model lists, BYOK availability badges, auto-routing denials, Organization Auto targets, and the Cloud Agent $1 balance-check bypass all stop counting a Vercel provider key that routing would skip. Organization Auto also accepts a target that the default policy explicitly grants (section 2, step 3).
Behavior change for group provider grants
Before, an organization's Vercel provider key served every member who could use the model. Granting a provider to a group only made more models available to its members; it never decided who could use the key. After this PR, a group's
provider_allow_listalso decides which members use the organization's key for that provider. A member whose groups don't grant the key's provider skips the key. That member's request then takes the next route, usually Kilo's route billed to the organization's Kilo credits, or is denied. This is intended, since policies apply whoever pays for inference. Admins who want the key used by the whole organization add its provider to the organization allow-list or to every relevant group.4. Examples (one organization Bedrock key; model
anthropic/claude-haiku-4.5, providers Anthropic / Amazon Bedrock / Google Vertex / Azure)all["amazon-bedrock"]["anthropic"]only: ["anthropic"], billed to the organization's Kilo credits.["deepinfra"](no provider serves the model)model_not_allowednone, groupselectedwithprovider_allow_list: ["anthropic"]only: ["anthropic"].none, groupselectedwithmodel_allow_list: ["anthropic/claude-haiku-4.5"]none, no group grants the modelmodel_not_allowedmodel_not_allowedmodel_deny_list, baseline grantmodel_not_allowedmodel_not_allowed["anthropic"], Moonshot AI key, modelmoonshotai/kimi-k2.6(the #9092 case)model_not_allowedunlessmoonshotaiis added to the ceilingStill exempt (out of scope)
model_allow_listhas no provider limit on any route. This is a policy rule, not specific to BYOK.Verification
apps/web:llm-proxy,embeddings,byok/*,lib/organizations,auto-routing-denied-models, andai-gateway/providers/*. After merging the latestmainand addressing review: 108 suites passed (2674 tests, 2 skipped). The run also coveredai-gateway/auto-model,src/routers/organizationsandsrc/lib/cloud-agent-next/balance-check-eligibility.test.ts. The two review regressions (decrypt-failure fail-closed, and an Organization Auto default explicit grant) fail against the previous source and pass now. New or updated tests cover the helper, routing fall-through inget-provider.preview.test.ts(real DB keys), embedding provider selection, handler decisions (allowed key served, disallowed key denied, allow-list passed to routing), and availability.pnpm --filter @kilocode/web-shared typecheck,pnpm --filter ai-gateway typecheck,pnpm --filter web typecheck: pass.oxlintandoxfmton the changed files: clean.pnpm dependency-cycle-check: no violations.Live E2E (local stack, this branch)
Run before the latest
mainmerges. The merged changes (BYOK KMS retries, provider listing, Anthropic headers) do not touch the routing or policy code exercised here. The review fixes only change which keys are decrypted (now only allowed ones) and Organization Auto validation. These results therefore still apply.Setup: an isolated worktree stack, with the provider and direct-BYOK catalogs synced from the cron routes. A dedicated test user owns Enterprise orgs that each hold one organization Bedrock key (real credentials) and differ only in
provider_allow_list. Each request is oneanthropic/claude-haiku-4.5chat completion withmax_tokens: 8.["amazon-bedrock"]vercel/bedrockwith BYOK (usage row cost 0)["anthropic"](model allowed, key's provider not)only: ["anthropic"], which the local OpenRouter account rejected (404 "All providers have been ignored"). The key was never used["deepinfra"](no allowed provider serves the model)model_not_allowed, no upstream requestWith a zero org balance, the last two returned 402 credits-blocked instead, which also confirms the BYOK key was not used.
Limitations: embeddings were checked with unit tests only, not live. I did not re-run a live pre-fix baseline. The bypass is shown by the code path described in Problem.
Models: Claude Opus 5.5 (no subagents).