[chrarnoldus]

Summary

kilocode-ai-gateway runs in fra1 and sfo1, but db always connects to the Frankfurt primary. Several reads that happen before the upstream inference request used db, so sfo1 requests paid transatlantic round trips before the provider was called. These now use readDb, which is the US replica in sfo1 (POSTGRES_REPLICA_US_URL is set on the gateway's production project).

• Org model-access policy (every org request): getOrganizationGroupPolicyContext ran a read-only repeatable read transaction on the primary: BEGIN, up to four SELECTs, COMMIT. It now takes an optional fromDb. Gateway callers pass readDb: LLM proxy, FIM, edit, embeddings, audio, systemone, models, models-by-provider, embedding-models, transcription-models, and the auto-routing denied-models route. Web callers keep the primary default.
• Balance lookups: embeddings, audio transcriptions and systemone called getBalanceAndOrgSettings without a client, so the org balance row was read from the primary. They now pass readDb, as the LLM proxy, FIM and edit already did.
• ChatGPT BYOK: the eligibility read and the token fast path use the replica. A lagging row can only return a token that was already usable without refresh. Refresh still re-reads the row on the primary under FOR UPDATE.
• 60 s per-instance config caches (unavailable models, routing config, auto-free config, preferred models) and the 10 s request-logging opt-in cache read the replica. On cache expiry the refresh is awaited inline, so a primary read there stalled requests. The admin opt-in list still reads the primary.
• Guidance: packages/web-shared/AGENTS.md has a new "AI gateway database access" section: no primary reads, transactions, locks or writes before inference; pass readDb explicitly because helpers default to db; keep POSTGRES_REPLICA_US_URL set. apps/ai-gateway/AGENTS.md points to it. ENVIRONMENT.md now documents the replica variables.

Not changed, and intentionally still on the primary before inference:

• Lazy credit expiry: optimistic writes, about once per expiry event per account.
• ChatGPT token refresh: row lock and write, about once per token lifetime.
• isAutoTopUpInFlight: read-after-write, and only on the out-of-credits path, which returns an error and never reaches inference.

Verification

• Checked with vercel env ls that POSTGRES_REPLICA_US_URL is set for production on kilocode-ai-gateway.
• No manual request testing. Local development has no separate replica (readDb is the primary in development), so the latency effect can only be observed in sfo1 production traces.

Visual Changes

N/A

Reviewer Notes

• Replica lag (normally under 100 ms) is the accepted trade-off. A policy, ChatGPT connection, or config change can take that long longer to apply in the gateway. Config reads were already cached for 10–60 s.
• In fra1, readDb now spreads these reads across the EU replicas instead of the primary.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The replica-routing change is consistent: every gateway pre-inference caller of the org policy, balance, ChatGPT connection, and gateway config helpers now passes readDb, shared helpers keep their primary default, and primary-only paths (row locks, token refresh, writes, admin reads) remain on db.

Files Reviewed (31 files)

• ENVIRONMENT.md
• apps/ai-gateway/AGENTS.md
• apps/web/src/tests/openrouter-models.test.ts
• packages/web-shared/AGENTS.md
• packages/web-shared/src/lib/ai-gateway/auto-model/auto-free-config.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-denied-models.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-routing-denied-models.ts
• packages/web-shared/src/lib/ai-gateway/handlers/audio-transcriptions.ts
• packages/web-shared/src/lib/ai-gateway/handlers/edit-completions.ts
• packages/web-shared/src/lib/ai-gateway/handlers/embedding-models.ts
• packages/web-shared/src/lib/ai-gateway/handlers/embeddings.ts
• packages/web-shared/src/lib/ai-gateway/handlers/fim-completions.ts
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models-by-provider.ts
• packages/web-shared/src/lib/ai-gateway/handlers/models.ts
• packages/web-shared/src/lib/ai-gateway/handlers/transcription-models.ts
• packages/web-shared/src/lib/ai-gateway/openai-chatgpt/refresh.test.ts
• packages/web-shared/src/lib/ai-gateway/openai-chatgpt/refresh.ts
• packages/web-shared/src/lib/ai-gateway/openai-chatgpt/store.test.ts
• packages/web-shared/src/lib/ai-gateway/openai-chatgpt/store.ts
• packages/web-shared/src/lib/ai-gateway/preferred-models.ts
• packages/web-shared/src/lib/ai-gateway/providers/routing-config.ts
• packages/web-shared/src/lib/ai-gateway/request-logging-opt-ins.ts
• packages/web-shared/src/lib/ai-gateway/typesafe/handler.test.ts
• packages/web-shared/src/lib/ai-gateway/typesafe/handler.ts
• packages/web-shared/src/lib/ai-gateway/unavailable-models.ts
• packages/web-shared/src/lib/organizations/group-policies/model-access/model-access.server.ts
• packages/web-shared/src/lib/organizations/organization-group-policy-context.server.ts
• packages/web-shared/src/lib/organizations/organization-models.test.ts
• packages/web-shared/src/lib/organizations/organization-models.ts

---

_{Reviewed by deepseek-v4.1-flash-1017 · Input: 118.9K · Output: 35K · Cached: 3.9M}

_{Review guidance: REVIEW.md from base branch main}