Skip to content

fix(byok): show BYOK only for keys a shared-services run can use - #7819

Merged
iscekic merged 3 commits into
mainfrom
fix/code-review-byok-badge
Oct 19, 2026
Merged

iscekic merged 3 commits into
mainfrom
fix/code-review-byok-badge

Conversation

@iscekic

@iscekic iscekic commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The Code Reviewer model picker showed an OpenAI model as BYOK when one organization member had connected their own ChatGPT account. A review runs as the organization's bot, and the gateway serves a bot run from the organization's shared-services ChatGPT connection only (openai-chatgpt/routing.ts, openAiChatGptOwnerCandidates). It never uses a member's own connection. The review then fell back to Kilo-billed routing, and the customer was charged for reviews they believed were BYOK. The same wrong badge appeared in every picker whose model a platform service runs for the organization.

Fix

  • One server switch. getAvailableModelsForOrganization and organizations.settings.listAvailableModels take an optional audience: member (the default) or shared_services. A shared_services list tags ChatGPT-served models only from openAiChatGptSharedServicesOwner(organizationId). Organization BYOK API keys tag as before. GET /api/organizations/{id}/models?audience=shared_services passes it through and rejects an unknown value with 400.
  • The contract lives in @kilocode/app-shared/organizations (ORGANIZATION_MODEL_AUDIENCES, organizationModelsPath). Web and mobile use it to build the URL.
  • Members' own interactive pickers keep the member list and their personal-connection badge.

Pickers switched to shared_services

Each one configures a run whose gateway token carries a bot id and an organization:

Surface Web Mobile
Code Reviewer (GitHub / GitLab config) ReviewConfigForm platform-overview-screen
Code Reviewer (Bitbucket config) BitbucketReviewConfigForm bitbucket-overview
Code Reviewer manual review CodeReviewJobsCard manual-review-screen
Auto-fix AutoFixConfigForm — (no picker)
Auto-triage AutoTriageConfigForm — (no picker)
Webhook triggers (create / edit) CreateWebhookTriggerContent, EditWebhookTriggerContent — (no picker)
Discord bot DiscordIntegrationDetails — (no picker)

Mobile requests the list through the new useReviewerModels(scope) hook. Mobile computes no badge of its own: it renders the server's hasUserByokAvailable.

Deliberately unchanged

  • Personal (non-organization) variants of these surfaces. A bot token without an organization uses the caller's own personal connection, and those pickers already read the personal list (/api/openrouter/models). The badge was already correct there.
  • Security agent, Slack, Linear, GitHub-App bot, KiloClaw, Gastown, App Builder, cloud agent sessions, chat. Their tokens carry no bot id, so the gateway uses the member's own connection, and the member list is correct for them.

Verification

  • organization-models.test.ts: a new regression test. A member-only ChatGPT connection tags the model in the member list but not in the shared-services list. A shared-services connection tags it in the shared-services list. Organization BYOK keys still tag the shared-services list. With the producer change stashed, the test fails.
  • handlers/organization-models.test.ts: the audience query parameter reaches the catalog, and an unknown value returns 400.
  • Type checks pass for app-shared, web-shared, web, ai-gateway and mobile. Mobile lint and check:unused pass, and the mobile code-reviewer and hooks suites pass.
  • End to end on a disposable VM, with a local web app and gateway, a fake organization, and a stub OpenAI upstream. No real accounts or secrets were used. Screenshots are in the PR comment.
    • Only a member has a ChatGPT connection: the member list tags openai/gpt-5.6-terra BYOK, but the shared-services list does not, and the Code Reviewer picker shows no BYOK badge. A request made with the review bot's token goes to the Kilo-billed route, and the stub receives nothing.
    • A shared-services connection is added: the shared-services list tags the model, and the picker shows the BYOK badge. The same bot request reaches the stub through the shared-services connection, and the usage row records provider openai-chatgpt.
  • CI passes on the latest head.

iscekic and others added 2 commits October 17, 2026 04:23
Model pickers for runs the platform makes for an organization (code
review, auto-fix, auto-triage, webhook triggers, Discord bot) badged
models BYOK from the viewing member's personal ChatGPT connection. The
gateway never uses that connection for a service run, so those runs were
billed by Kilo.

The organization model list now takes an audience. A shared_services
list tags ChatGPT models only from the organization's shared-services
connection; organization BYOK keys tag as before. Members' own pickers
keep the member list. Web and mobile pickers for service runs request
the shared_services list.
@kilo-code-bot

kilo-code-bot Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental changes only make the tRPC audience input default to member at the schema boundary and align the integration-test expectation and generated catalog with that default; behavior is unchanged for callers that omit the parameter.

Files Reviewed (3 files)
  • apps/ai-gateway/src/app/api/v1/model-validation.integration.test.ts
  • packages/web-shared/src/routers/organizations/organization-available-models-procedure.ts
  • services/kilo-mcp/catalog.json (generated)

Notes: verified listAvailableModelsProcedure now parses audience with .default('member'), which is equivalent to the producer's existing default in getAvailableModelsForOrganization; callers that omit the field (e.g. organization-model-validation.ts:31, web hooks.ts:620) still type-check and behave identically. The updated integration test expectation matches the real call path through the tRPC caller. No memory-leak or resource-lifetime concerns are introduced by these changes.

Previous Review Summary (commit 1ecc142)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 1ecc142)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The new audience switch is threaded consistently from the shared contract through the tRPC procedure, the HTTP handler, and every web/mobile picker that configures an organization service run; the OpenAI ChatGPT BYOK owner is now correctly scoped to the organization's shared-services connection, with no logic, security, or type defects found in the changed lines.

Files Reviewed (25 files)
  • apps/mobile/src/components/code-reviewer/bitbucket-overview.tsx
  • apps/mobile/src/components/code-reviewer/manual-review-screen.mounted.test.tsx
  • apps/mobile/src/components/code-reviewer/manual-review-screen.tsx
  • apps/mobile/src/components/code-reviewer/platform-overview-screen.mounted.test.tsx
  • apps/mobile/src/components/code-reviewer/platform-overview-screen.tsx
  • apps/mobile/src/lib/hooks/use-available-models.ts
  • apps/mobile/src/lib/hooks/use-reviewer-models.ts
  • apps/web/src/app/(app)/cloud/webhooks/[triggerId]/EditWebhookTriggerContent.tsx
  • apps/web/src/app/(app)/cloud/webhooks/new/CreateWebhookTriggerContent.tsx
  • apps/web/src/components/auto-fix/AutoFixConfigForm.tsx
  • apps/web/src/components/auto-triage/AutoTriageConfigForm.tsx
  • apps/web/src/components/cloud-agent/hooks/useOrganizationModels.ts
  • apps/web/src/components/code-reviews/BitbucketReviewConfigForm.tsx
  • apps/web/src/components/code-reviews/CodeReviewJobsCard.tsx
  • apps/web/src/components/code-reviews/ReviewConfigForm.tsx
  • apps/web/src/components/integrations/DiscordIntegrationDetails.tsx
  • apps/web/src/lib/hooks/use-openrouter-models.ts
  • packages/app-shared/src/organizations/index.ts
  • packages/app-shared/src/organizations/model-audience.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/organization-models.test.ts
  • packages/web-shared/src/lib/ai-gateway/handlers/organization-models.ts
  • packages/web-shared/src/lib/organizations/organization-models.test.ts
  • packages/web-shared/src/lib/organizations/organization-models.ts
  • packages/web-shared/src/routers/organizations/organization-available-models-procedure.ts
  • services/kilo-mcp/catalog.json (generated)

Notes: chatGptByokOwner matches the gateway's actual owner ordering — service runs are attributed to an org bot user, so only the shared-services connection can serve them, and the member list keeps its previous behavior. Generated catalog.json was checked only for schema shape: audience is optional and the entry remains a read-only query. No memory-leak or resource-lifetime issues are introduced by the hook signature/query-key changes.


Reviewed by deepseek-v4.1-flash · Input: 31.5K · Output: 4.5K · Cached: 309.4K

Review guidance: REVIEW.md from base branch main

@iscekic

iscekic commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

bot: End-to-end check on a disposable VM: local web app and ai-gateway, local Postgres, fake users, and a stub OpenAI upstream that logs every request. No real accounts or secrets were used.

Model: openai/gpt-5.6-terra. The stub's /models lists only this model.

Case 1: a member has a personal ChatGPT connection for the org. The org has no shared-services connection and no OpenAI BYOK key.

  • GET /api/organizations/<org>/models (member list): the model has hasUserByokAvailable: true.
  • GET /api/organizations/<org>/models?audience=shared_services: the same model has hasUserByokAvailable: false. No model in the list is tagged BYOK.
  • Code Reviewer model picker: no BYOK badge on the model.
  • Routing: I sent a responses request with a code-review bot token for the org. The stub got no request. The gateway sent the request to the Kilo-billed OpenRouter path, which rejected the fake key. The usage row has provider openrouter.
Case 1: no shared-services connection
Case 1

Case 2: I then added an org shared-services ChatGPT connection.

  • ?audience=shared_services now tags the model hasUserByokAvailable: true. The member list still tags it too.
  • Code Reviewer model picker: the model shows the BYOK badge. The -pro variants, which the stub does not serve, show no badge.
  • Routing: the same bot request returned 200 from the stub. The stub got one delegated POST /v1/responses. Its on-behalf-of token belonged to the shared-services connection, not the member's personal one. The usage row has provider openai-chatgpt and cost 0. It has is_user_byok = false, which is how the delegated route records usage today. This PR does not change that.
Case 2: shared-services connection present
Case 2

I also checked the badge state from the DOM. The BYOK badge is on the GPT-5.6 Terra option only in case 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants