[chrarnoldus]

Summary

Production query statistics showed four separate single-column reads of the singleton ai_gateway_config row (id = 1) among the most frequent gateway queries, despite 60 s in-memory caches:

Column	Reader
unavailable_models	ai-gateway/unavailable-models.ts
auto_free	ai-gateway/auto-model/auto-free-config.ts
preferred_models	ai-gateway/preferred-models.ts
config	ai-gateway/providers/routing-config.ts

Vercel function instances are short-lived, so each instance refreshes its in-memory caches often, and each refresh was a separate blocking query.

This PR:

• Adds ai-gateway-config-snapshot.ts, one cached snapshot of all four raw columns:
  • in-memory cache (60 s, unchanged) via createCachedFetch, with last-known-good on failure;
  • on a memory miss, GET ai-gateway-config:v1 from Upstash (us-west read replica, so local from sfo1), validated with zod. An invalid value is removed with a compare-and-delete script, so the populate can replace it without deleting a valid value a writer stored in between;
  • on a Redis miss, invalid value, or Redis error, one readDb select of all four columns, then a background SET NX EX 300 registered with waitUntil, which never blocks the response and never throws.
• Derives the four existing getters from the snapshot via createAiGatewayConfigSelector, which re-parses only when the snapshot object changes and keeps each getter's last valid value when its column is invalid. Each module keeps its parsing, validation, defaults, warnings, and exported API (readUnavailableModelIds, readPreferredModels, and readConfiguredAutoFreeModels still do an uncached replica read).
• Adds publishAiGatewayConfigSnapshot(), which every admin write calls after its upsert commits: it reads the full row back from the primary and SETs it with EX 300 (no NX). If the read-back fails, Redis is left untouched. If the SET fails, it deletes the key and logs; the TTL bounds the rest. It never throws.

Single-flight for concurrent in-memory misses is not part of this PR: #7881 (merged) added it, with a 30 s failure backoff, to createCachedFetch, which this snapshot cache uses.

Expected effect: an instance memory miss usually becomes one Redis read. The replica is only queried by instances that miss Redis during the short window after the key expires (300 s), or while Redis is unavailable. Each of those instances runs one query, where before every instance ran four per minute, multiplied by concurrent requests.

Verification

• NODE_ENV=test pnpm exec jest -i --forceExit (from apps/web, against this worktree's own Postgres container on port 10832 via apps/web/.env.test.local) on the new and directly changed suites: ai-gateway-config-snapshot.test.ts, cached-fetch.test.ts, unavailable-models.test.ts, preferred-models.test.ts, auto-free-config.test.ts, auto-free-config-router.test.ts, gateway-model-lists-router.test.ts, gateway-config-router.test.ts: all pass

• Same command with --runTestsByPath on 90 suites: every test file that imports createCachedFetch or one of its users, the four gateway config getters, or their consumers (gateway handlers, providers, auto-model resolution, sync crons, models routers, apps/ai-gateway route tests). 90 suites, 2323 tests passed, no unhandled rejections in the log

• pnpm --filter @kilocode/web-shared --filter web --filter ai-gateway typecheck: passes

• pnpm exec oxlint --config .oxlintrc.json <18 changed files>: 0 warnings, 0 errors

• pnpm format:changed and git diff --check: clean

• After merging main and addressing review: snapshot, getter, and admin router suites (8 suites, 99 tests), then jest -i on packages/web-shared/src/lib/ai-gateway, apps/ai-gateway, and apps/web/src/routers/admin (147 suites, 2967 tests): all pass; typecheck and oxlint clean

• After merging main again (including fix(web-shared): share in-flight fetches and back off failures in createCachedFetch #7881): jest -i on packages/web-shared/src/lib/ai-gateway, apps/ai-gateway, apps/web/src/routers/admin, and cached-fetch.test.ts (157 suites, 3105 tests): all pass; typecheck clean

Visual Changes

N/A

Reviewer Notes

Writers. These are all the writes to ai_gateway_config in the repository (searched apps, packages, services, scripts, dev seed data):

• apps/web/src/routers/admin/gateway-config-router.ts, set (config)
• apps/web/src/routers/admin/auto-free-config-router.ts, set (auto_free)
• apps/web/src/routers/admin/gateway-model-lists-router.ts, set (preferred_models, unavailable_models)

All three now await publishAiGatewayConfigSnapshot() after the upsert. No Cloudflare Worker, script, or other service writes this table, so no writer is left to rely on the TTL alone. Manual SQL edits in production are only picked up when the Redis TTL expires (up to 300 s), plus up to 60 s in each instance's memory.

Race reasoning.

• A reader that misses Redis, reads a lagging replica, and populates with SET NX cannot overwrite a writer's value. If the writer's SET lands first, the reader's NX write is a no-op. If the reader's lands first, the writer's plain SET overwrites it.
• Writers never use DEL for invalidation, because a reader could repopulate from a lagging replica right after it. DEL is only the fallback when the writer's SET fails.
• Two concurrent admin writes can finish their read-back-then-SET out of order, leaving the older full row in Redis for up to 300 s. Admin edits to this row are rare and done by hand, so I did not add versioning.
• Upstash replicates to its read replicas asynchronously, so a sfo1 reader can briefly see the previous value after a successful writer SET.

Staleness bound.

• After a successful admin write: up to 60 s per instance, the in-memory TTL. This matches the behavior before this PR.
• If the writer's SET fails but DEL succeeds: a reader can repopulate from a lagging replica. That stale value lasts at most 300 s in Redis plus 60 s in memory, and in practice only as long as the replica lag.
• If both fail: up to 300 s plus 60 s.

Other notes.

• When no config row exists, the snapshot is null and is not written to Redis. The getters return their existing defaults, as before.
• The Redis zod schema checks only the snapshot's shape: four JSON values, all present. Each column is still validated by its own module, so an invalid stored column keeps that getter's previous behavior: last valid value or default, and auto_free still warns and falls back to null. It does not make the whole snapshot uncacheable.
• The background SET NX uses waitUntil from @vercel/functions rather than after(), because the snapshot load is shared across requests and can also run outside a request scope (crons, scripts), where waitUntil is a no-op and the write still runs.
• Gateway paths still never touch the primary before inference. Only the admin writer helper reads db.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

Re-reviewed the hardening commit be8f4945d (separated primary read-back from the Redis write, compare-and-delete repair of invalid cached values, waitUntil-registered populate, and removal of the createCachedFetch change); no new defects were found in the changed lines.

Files Reviewed (18 files)

• apps/web/src/routers/admin/auto-free-config-router.test.ts
• apps/web/src/routers/admin/auto-free-config-router.ts
• apps/web/src/routers/admin/gateway-config-router.test.ts
• apps/web/src/routers/admin/gateway-config-router.ts
• apps/web/src/routers/admin/gateway-model-lists-router.test.ts
• apps/web/src/routers/admin/gateway-model-lists-router.ts
• packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.test.ts
• packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.ts
• packages/web-shared/src/lib/ai-gateway/auto-model/auto-free-config.test.ts
• packages/web-shared/src/lib/ai-gateway/auto-model/auto-free-config.ts
• packages/web-shared/src/lib/ai-gateway/preferred-models.test.ts
• packages/web-shared/src/lib/ai-gateway/preferred-models.ts
• packages/web-shared/src/lib/ai-gateway/providers/routing-config.ts
• packages/web-shared/src/lib/ai-gateway/unavailable-models.test.ts
• packages/web-shared/src/lib/ai-gateway/unavailable-models.ts
• packages/web-shared/src/lib/cached-fetch.test.ts
• packages/web-shared/src/lib/cached-fetch.ts
• packages/web-shared/src/lib/redis-keys.ts

Previous Review Summary (commit bead522)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit bead522)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

The snapshot design is sound, but the new single-flight helper has a latent logic bug that can permanently disable refreshes, and the publish fallback deletes the shared key on a read-back failure, which can extend staleness to roughly 6 minutes.

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	2

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/cached-fetch.ts	30	A synchronously-throwing fetcher leaves inFlight set to an already-resolved promise, so the cache never refreshes again. Latent for current async callers, but shared helper.
packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.ts	130	The catch covers the primary read-back, so a read failure deletes the shared key and lets a lagging replica repopulate it with a fresh 300 s TTL; it can also delete a concurrent publisher's newer value.

SUGGESTION

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.ts	50	An invalid cached value is left in place while the NX populate cannot replace it, so every memory miss repeats the warning and replica read until the TTL expires.
packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.ts	74	The fire-and-forget Redis populate is not tied to waitUntil/after(), so a serverless invocation may drop it and lose the intended cache warmup.

Files Reviewed (18 files)

• apps/web/src/routers/admin/auto-free-config-router.test.ts - 0 issues
• apps/web/src/routers/admin/auto-free-config-router.ts - 0 issues
• apps/web/src/routers/admin/gateway-config-router.test.ts - 0 issues
• apps/web/src/routers/admin/gateway-config-router.ts - 0 issues
• apps/web/src/routers/admin/gateway-model-lists-router.test.ts - 0 issues
• apps/web/src/routers/admin/gateway-model-lists-router.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/ai-gateway-config-snapshot.ts - 3 issues
• packages/web-shared/src/lib/ai-gateway/auto-model/auto-free-config.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/auto-model/auto-free-config.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/preferred-models.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/preferred-models.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/providers/routing-config.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/unavailable-models.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/unavailable-models.ts - 0 issues
• packages/web-shared/src/lib/cached-fetch.test.ts - 0 issues
• packages/web-shared/src/lib/cached-fetch.ts - 1 issue
• packages/web-shared/src/lib/redis-keys.ts - 0 issues

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-v4.1-flash-1017 · Input: 101.4K · Output: 22.3K · Cached: 2.2M}

_{Review guidance: REVIEW.md from base branch main}