Skip to content

[AISAGE-857] fix(byok): enforce the organization provider allow-list on Vercel provider keys - #9215

Merged
chrarnoldus merged 6 commits into
mainfrom
fix/user-byok-enterprise-provider-policy
Oct 30, 2026
Merged

chrarnoldus merged 6 commits into
mainfrom
fix/user-byok-enterprise-provider-policy

Conversation

@chrarnoldus

@chrarnoldus chrarnoldus commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Part of AISAGE-857: apply organization model policies to user BYOK.

Problem

In an organization context, a provider key routed through Vercel (for example Bedrock, Anthropic, Mistral, or Moonshot AI) bypassed the Enterprise provider allow-list. applyVercelSettings pinned every organization key and dropped the policy's only list. allowsInferenceProviderUserByok also admitted organization_provider denials for those keys. The result was that a prompt could reach a provider the admin had not allowed. For example, with an org allow-list of ["deepinfra"] and a Bedrock key, anthropic/claude-haiku-4.5 was served by Bedrock.

#9092 added this exemption on purpose ("the provider itself is considered trusted"). This PR changes that decision for Vercel provider keys. The provider allow-list is now enforced whenever it can name the key's provider, meaning the Vercel provider maps to an OpenRouter provider slug on the provider policy screen. Today that covers every Vercel BYOK provider. Direct BYOK keys, catalog mode included, are unchanged. No active production organization is affected: a production query for Enterprise organizations with an enabled Vercel provider key outside an org-wide or selected policy provider allow-list found none with recent activity. The trade-off: the #9092 case (allow-list ["anthropic"] plus a Moonshot AI key for Kimi) is denied again unless the org adds moonshotai to the allow-list.

Fix

  • New isVercelUserByokProviderAllowed(providerId, allowedProviders): a Vercel key is admitted when the policy doesn't restrict providers, when the policy can't name the key's provider, or when an allowed OpenRouter slug maps to it (amazon-bedrock→bedrock, google-vertex→vertex, google-ai-studio→google, z-ai→zai; a codestral key counts as mistral).
  • getProvider gets a new getAllowedProviders input. Providers outside the allow-list are dropped before their keys are loaded or decrypted, so the request falls through to the next route: a gateway key, or Kilo's gateway with the allow-list applied. Because disallowed keys are never decrypted, an allowed key that fails to decrypt still fails the request closed instead of falling back to Kilo credits.
  • llm-proxy supplies the member's eligibleProviderRoutes, or an empty set for an organization_provider denial. It also rejects with model_not_allowed, as defence in depth, if a Vercel route still carries a disallowed key.
  • Embeddings: the member decision is now resolved before provider selection, and its routes are passed to getEmbeddingProvider.
  • Model listing, BYOK availability badges, auto-routing denials, and Organization Auto (hasInferenceProviderUserByokForModel, hasEnabledUserByokForModel) apply the same rule, so they no longer offer models that requests would reject.
  • Organization Auto target validation: when the raw provider allow-list rejects a target, the effective default model_access policy is evaluated with the settings being saved. A model that the default policy explicitly grants stays a valid target, matching request-time behavior. This replaces the Moonshot-key admission that the new rule removes.
  • Cloud Agent balance check (computeCloudAgentNextBalanceCheckEligibility) resolves the member decision for every organization request. A provider key that routing would drop no longer skips the $1 minimum.
  • Updated the comments and the BYOK dialog text, which previously called the provider "trusted".

Manual: how policies affect models and providers

Private custom LLMs are out of scope. "Before" means main without this PR, which already includes #9092.

1. Which policy applies

Account Policy
Personal account None. Every key and route can be used.
Teams organization No model or provider restrictions. Only the data-collection setting applies.
Enterprise organization The organization baseline plus group model_access policies, combined as below.

In an organization, routing uses only keys the organization stored. A member's personal keys are never used, so they can't get around the policy.

2. Who gets what in an Enterprise organization

Inputs:

  • The organization baseline: model_deny_list and provider_allow_list (the provider "ceiling").
  • The default model_access policy, plus the policy of every group the member belongs to. Each policy is all, none, or selected, where selected has model_allow_list and provider_allow_list.

The member's grant:

  • Baseline only: applies when there is no default policy, or every policy is all, and no policy is selected.
  • Selected: applies when any policy is selected. The member gets the union of every selected model_allow_list and provider_allow_list. The baseline is added on top when there is no default policy, the default is all, or any group policy is all.
  • No grant: applies when every policy is none, or the default is none and no group policy grants anything.

Decision for one model, which yields allowed or denied plus a set of allowed provider routes. Provider routes are OpenRouter provider slugs: the model's providers in the current snapshot, intersected with the allowed providers.

  1. Kilo Auto ids, and bespoke direct-BYOK ids such as neuralwatt/..., are exempt. They are allowed with no provider routes.
  2. Baseline grant: a model on the deny list is denied (organization_model). A model not in the snapshot is denied. Without a ceiling, the model is allowed with no route limit. With a ceiling, the routes are the model's providers that fall within the ceiling. An empty result is denied as organization_provider.
  3. Selected grant:
    • A model named in a selected model_allow_list is allowed with no route limit. Neither the organization deny list nor the ceiling applies to it.
    • Otherwise, when the model's providers overlap the selected provider_allow_list, it is allowed with those providers as routes. If the baseline is included, the baseline's routes are added. A baseline with no route limit lifts the limit.
    • Otherwise, the baseline decides when it is included, as in step 2.
    • Otherwise the model is denied (no_grant).

organization_provider is the only denial reason that means "the model is allowed, but none of its providers are". A group's provider list never produces that reason: when a group's providers don't serve the model, the result is no_grant.

3. How each route applies the decision

"Routes" below means the allowed provider routes from section 2.

Route (in getProvider order) Model decision Provider routes, before Provider routes, after
Bespoke direct-BYOK id (provider/model naming a direct provider) Exempt Ignored Ignored (unchanged)
Catalog-mode direct-BYOK key Applies. An organization_provider denial is admitted. Ignored, because the request goes straight to the key's provider Ignored (unchanged)
Personal "Sign in with ChatGPT" (eligible /responses) Applies Enforced (no admission) Enforced (unchanged)
Vercel provider key (anthropic, bedrock, vertex, openai, mistral/codestral, moonshotai, ...) Applies. Before, an organization_provider denial was admitted. Ignored: every organization key was sent to Vercel, including keys outside the routes. Enforced: a key is used only when an allowed route maps to its provider (amazon-bedrock→bedrock, google-vertex→vertex, google-ai-studio→google, z-ai→zai, mistral→mistral/codestral). Other keys are skipped and the next route takes the request. An organization_provider denial leaves no allowed provider, so every such key is skipped. A provider the policy screen cannot name would stay exempt, but none exists today.
Gateway key: Vercel AI Gateway Applies Used only if Vercel can serve the model within the routes; otherwise falls through to the OpenRouter key Unchanged
Gateway key: OpenRouter Applies Request sent with only: routes Unchanged
Kilo route (Vercel or OpenRouter, paid by Kilo) Applies Request sent with only: routes Unchanged. It now also receives requests whose Vercel provider keys were skipped.
Public custom LLM Applies Declared providers intersected with the routes and the caller's lists Unchanged

Embeddings use the same decision. Before, a Vercel provider key was chosen before the decision was evaluated, and the routes were never applied to it. After, the routes filter keys in the same way, and a skipped key falls through to OpenRouter with only: routes.

The same rule is used wherever the product predicts routing. Model lists, BYOK availability badges, auto-routing denials, Organization Auto targets, and the Cloud Agent $1 balance-check bypass all stop counting a Vercel provider key that routing would skip. Organization Auto also accepts a target that the default policy explicitly grants (section 2, step 3).

Behavior change for group provider grants

Before, an organization's Vercel provider key served every member who could use the model. Granting a provider to a group only made more models available to its members; it never decided who could use the key. After this PR, a group's provider_allow_list also decides which members use the organization's key for that provider. A member whose groups don't grant the key's provider skips the key. That member's request then takes the next route, usually Kilo's route billed to the organization's Kilo credits, or is denied. This is intended, since policies apply whoever pays for inference. Admins who want the key used by the whole organization add its provider to the organization allow-list or to every relevant group.

4. Examples (one organization Bedrock key; model anthropic/claude-haiku-4.5, providers Anthropic / Amazon Bedrock / Google Vertex / Azure)

Policy Before After
Personal account or Teams organization Bedrock key Bedrock key
Enterprise, no ceiling, default all Bedrock key Bedrock key
Ceiling ["amazon-bedrock"] Bedrock key Bedrock key
Ceiling ["anthropic"] Bedrock key Key skipped. Kilo route with only: ["anthropic"], billed to the organization's Kilo credits.
Ceiling ["deepinfra"] (no provider serves the model) Bedrock key model_not_allowed
Default none, group selected with provider_allow_list: ["anthropic"] Bedrock key Key skipped. Kilo route with only: ["anthropic"].
Default none, group selected with model_allow_list: ["anthropic/claude-haiku-4.5"] Bedrock key Bedrock key (an explicit model grant has no provider limit)
Default none, no group grants the model model_not_allowed model_not_allowed
Model on model_deny_list, baseline grant model_not_allowed model_not_allowed
Ceiling ["anthropic"], Moonshot AI key, model moonshotai/kimi-k2.6 (the #9092 case) Moonshot key model_not_allowed unless moonshotai is added to the ceiling

Still exempt (out of scope)

  • Catalog-mode direct-BYOK keys: model rules apply, but the provider allow-list does not, because most direct providers cannot be named on the provider policy screen. This is a candidate follow-up.
  • Bespoke direct-BYOK ids: exempt from organization model and provider policy, as before.
  • A model named in a group model_allow_list has no provider limit on any route. This is a policy rule, not specific to BYOK.

Verification

  • Jest from apps/web: llm-proxy, embeddings, byok/*, lib/organizations, auto-routing-denied-models, and ai-gateway/providers/*. After merging the latest main and addressing review: 108 suites passed (2674 tests, 2 skipped). The run also covered ai-gateway/auto-model, src/routers/organizations and src/lib/cloud-agent-next/balance-check-eligibility.test.ts. The two review regressions (decrypt-failure fail-closed, and an Organization Auto default explicit grant) fail against the previous source and pass now. New or updated tests cover the helper, routing fall-through in get-provider.preview.test.ts (real DB keys), embedding provider selection, handler decisions (allowed key served, disallowed key denied, allow-list passed to routing), and availability.
  • pnpm --filter @kilocode/web-shared typecheck, pnpm --filter ai-gateway typecheck, pnpm --filter web typecheck: pass.
  • oxlint and oxfmt on the changed files: clean. pnpm dependency-cycle-check: no violations.

Live E2E (local stack, this branch)

Run before the latest main merges. The merged changes (BYOK KMS retries, provider listing, Anthropic headers) do not touch the routing or policy code exercised here. The review fixes only change which keys are decrypted (now only allowed ones) and Organization Auto validation. These results therefore still apply.

Setup: an isolated worktree stack, with the provider and direct-BYOK catalogs synced from the cron routes. A dedicated test user owns Enterprise orgs that each hold one organization Bedrock key (real credentials) and differ only in provider_allow_list. Each request is one anthropic/claude-haiku-4.5 chat completion with max_tokens: 8.

Org allow-list Result
["amazon-bedrock"] 200, real completion through vercel/bedrock with BYOK (usage row cost 0)
["anthropic"] (model allowed, key's provider not) Bedrock key skipped. Routed to Kilo's OpenRouter route with only: ["anthropic"], which the local OpenRouter account rejected (404 "All providers have been ignored"). The key was never used
["deepinfra"] (no allowed provider serves the model) 404 model_not_allowed, no upstream request

With a zero org balance, the last two returned 402 credits-blocked instead, which also confirms the BYOK key was not used.

Limitations: embeddings were checked with unit tests only, not live. I did not re-run a live pre-fix baseline. The bypass is shown by the code path described in Problem.

Models: Claude Opus 5.5 (no subagents).

@chrarnoldus chrarnoldus self-assigned this Oct 29, 2026
…ise-provider-policy

# Conflicts:
#	packages/web-shared/src/lib/ai-gateway/byok/index.ts
@kilo-code-bot

kilo-code-bot Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Both prior findings are resolved in current code: getGatewayRoutedByok now filters candidate providers through isVercelUserByokProviderAllowed before getBYOKforOrganization/getBYOKforUser loads or decrypts anything (get-provider.ts:412-429), so the loader's all-failed guard only sees eligible keys, and isAllowedAutoTarget now evaluates the effective default model_access policy with the plan and settings being validated (organization-auto-model.ts:104-147), so a default explicit grant stays a valid Organization Auto target.

Assumptions: [INFERENCE] no tests were executed (read-only review); the two fixes are verified by reading the current sources and the new regression tests (get-provider.preview.test.ts, organization-auto-model.test.ts). [INFERENCE] the default-policy branch adds a second organization policy context read on the rare path where the raw restrictions reject a target; it opens a read-only transaction against the caller's readDb/transaction, not the primary. [INFERENCE] an explicitly granted default policy admits the model for every member because group policies only add grants and cannot narrow the default policy's union.

Files Reviewed (4 files)
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.preview.test.ts - 0 issues
  • packages/web-shared/src/lib/organizations/organization-auto-model.ts - 0 issues
  • packages/web-shared/src/lib/organizations/organization-auto-model.test.ts - 0 issues
Previous Review Summaries (2 snapshots, latest commit 5887ccd)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 5887ccd)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change resolves the remaining finding: computeCloudAgentNextBalanceCheckEligibility now resolves the member model decision for every organization request and passes new Set() on a denial, matching llm-proxy, so a provider key that routing drops no longer skips the $1 minimum (verified against availability.ts:26-31, inference-provider-id.ts:290-303, and get-provider.ts:424-428).

Assumptions: [INFERENCE] prior finding at llm-proxy.ts:867 is dropped as by design: an allowed decision without eligibleProviderRoutes also leaves only unset for Kilo-funded routes (llm-proxy.ts:870-874), so Vercel provider keys follow the same effective policy. Prior suggestion at byok/index.ts:328 is dropped as intentional per the author's rationale. [INFERENCE] a cached Vercel provider list that is unavailable makes only: [] resolve to "compatible" (hasCompatibleVercelInferenceProvider returns true for null), a rare degraded-data case that can still skip the $1 minimum; not flagged as a finding. [INFERENCE] the new unconditional policy lookup adds one read-only transaction per organization session creation; accepted as the cost of correctness.

Files Reviewed (2 files)
  • apps/web/src/lib/cloud-agent-next/balance-check-eligibility.ts - 0 issues
  • apps/web/src/lib/cloud-agent-next/balance-check-eligibility.test.ts - 0 issues

Also re-checked for the carried-forward findings: packages/web-shared/src/lib/ai-gateway/byok/availability.ts, providers/get-provider.ts, providers/openrouter/inference-provider-id.ts, handlers/llm-proxy.ts, byok/index.ts, and group-policies/model-access/model-access.server.ts.

Previous review (commit 45684dc)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The new Vercel BYOK allow-list check is skipped when the model decision is allowed without eligibleProviderRoutes, and the availability filter is not wired into the cloud-agent balance-check caller.

Assumptions: [INFERENCE] the same eligibleProviderRoutes gap likely applies to the embeddings handler (embeddings.ts:225), which passes the routes directly. [INFERENCE] a provider that Vercel lists for a model but the OpenRouter snapshot does not may be falsely denied, because eligibleProviderRoutes is intersected with the OpenRouter provider list.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts 867 allowedProviders is undefined when an allowed decision omits eligibleProviderRoutes (selected group model grant, restriction-exempt models), and the helper treats undefined as unrestricted, so Vercel keys outside the org provider allow-list are admitted.
packages/web-shared/src/lib/ai-gateway/byok/availability.ts 28 The new provider-key filter is inert in computeCloudAgentNextBalanceCheckEligibility, which resolves allowedProviders only for gateway keys, so a disallowed provider key still reports BYOK availability and skips the worker-side $1 balance minimum.

SUGGESTION

File Line Issue
packages/web-shared/src/lib/ai-gateway/byok/index.ts 328 With NO_ALLOWED_PROVIDERS, the Vercel branch is unreachable because every derived provider id is nameable; the docstring's "provider the allow-list cannot name" case cannot occur.
Files Reviewed (15 files)
  • apps/web/src/components/organizations/byok/BYOKKeysManager.tsx - 0 issues
  • packages/web-shared/src/lib/ai-gateway/byok/availability.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/byok/availability.ts - 1 issue
  • packages/web-shared/src/lib/ai-gateway/byok/index.ts - 1 issue
  • packages/web-shared/src/lib/ai-gateway/handlers/embeddings.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/handlers/embeddings.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts - 1 issue
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.preview.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/getEmbeddingProvider.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.test.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.ts - 0 issues
  • packages/web-shared/src/lib/ai-gateway/providers/vercel/index.ts - 0 issues
  • packages/web-shared/src/lib/organizations/group-policies/model-access/model-access.server.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-flash-latest · Input: 89.1K · Output: 19.9K · Cached: 1.7M

Review guidance: REVIEW.md from base branch main

@chrarnoldus chrarnoldus left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The new filtering can silently switch a request with an undecryptable eligible BYOK key to organization-credit billing, and can reject Organization Auto targets explicitly permitted by the default model-access policy.

Assumptions: none. Both findings were reproduced using actual revision sources with mocked database/catalog/KMS boundaries. Full local suites and live-provider tests were not rerun; CI is green at the reviewed head.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
packages/web-shared/src/lib/ai-gateway/providers/get-provider.ts 424-426 [P2] Post-decryption policy filtering can defeat the credential loader's fail-closed billing protection.
packages/web-shared/src/lib/ai-gateway/byok/index.ts 326-330 [P2] Organization Auto rejects targets explicitly granted by the default model-access policy.
Files Reviewed (17 files)
  • apps/web/src/components/organizations/byok/BYOKKeysManager.tsx
  • apps/web/src/lib/cloud-agent-next/balance-check-eligibility.test.ts
  • apps/web/src/lib/cloud-agent-next/balance-check-eligibility.ts
  • packages/web-shared/src/lib/ai-gateway/byok/availability.test.ts
  • packages/web-shared/src/lib/ai-gateway/byok/availability.ts
  • packages/web-shared/src/lib/ai-gateway/byok/index.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/embeddings.test.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/embeddings.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.preview.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/get-provider.ts
  • packages/web-shared/src/lib/ai-gateway/providers/getEmbeddingProvider.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.test.ts
  • packages/web-shared/src/lib/ai-gateway/providers/openrouter/inference-provider-id.ts
  • packages/web-shared/src/lib/ai-gateway/providers/vercel/index.ts
  • packages/web-shared/src/lib/organizations/group-policies/model-access/model-access.server.ts

@chrarnoldus chrarnoldus changed the title fix(byok): enforce the organization provider allow-list on Vercel provider keys [AISAGE-857] fix(byok): enforce the organization provider allow-list on Vercel provider keys Oct 30, 2026
@chrarnoldus
chrarnoldus added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit ce5af00 Oct 30, 2026
57 checks passed
@chrarnoldus
chrarnoldus deleted the fix/user-byok-enterprise-provider-policy branch October 30, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants