[chrarnoldus]

Summary

Follow-up to #8312. The free-model-inference firewall rule counted every free-model request per IP (ip:<ip>), including requests from signed-in users. That blocked signed-in users who share an IP (NAT, offices), while heavy free use already moves across many IPs. Accounts are fingerprinted at sign-up and can be blocked; IPs are cheap to rotate.

• Signed-in callers now count against a new per-account rule, free-model-inference-user, keyed on user:<kiloUserId>. The account comes from the resolved auth, which has already resolved by this point, so the check adds no query. Session-cookie callers count per account too. A rejected credential, including a signature-valid but revoked token, gets the invalid-token response instead of spending the account's allowance or getting the sign-up prompt. This includes server-side products (cloud-agent, code-review, app-builder, gastown), which always authenticate.
• Anonymous callers keep the per-IP free-model-inference rule. When they hit it, the gateway again returns error.code: "PROMOTION_MODEL_LIMIT_REACHED" with status 429. [AISAGE-729] refactor(gateway): replace free model database rate limit with Vercel Firewall rule #7534 removed this response. Kilo clients still handle it: VS Code shows a sign-up prompt, the CLI shows a sign-up toast, and neither retries the request.
• Signed-in callers who hit their limit get a 429 that points to paid models or their own API key, since a sign-up prompt would be wrong for them.
• Both limit responses send x-should-retry: false. The windows are an hour long, so an immediate retry only spends more of the allowance. Kilo CLI v2 and the OpenAI SDKs honor the header.
• Removed the 401 for anonymous server-side product requests on Cloudflare IPs. It protected nothing: the feature header is client-supplied, and omitting it already led to the per-IP path. Renamed isUserRateLimitedFeature to isServerSideProductFeature; its only remaining use is keeping shared Cloudflare egress IPs out of bouncer's payer-sharing signal.

Client handling

Kilo clients on Kilo-Org/kilocode main:

• VS Code: isUnauthorizedPromotionLimitError accepts the code with status 401 or 429, and ErrorDisplay renders the sign-up card.
• CLI: parseKiloErrorCode reads error.code (or a top-level code), the TUI shows the sign-up toast, and retryable does not retry Kilo error codes. The signed-in 429 has no Kilo code, so it is still retried with backoff.

Kilo v2 (kilo-v2):

• The core retry policy retries RateLimit errors but honors x-should-retry, hence the header on both limit responses.
• Sessions report structured errors (type, message, status) without the response body. The webview's promotion check expects a v1 APIError, so v2 probably shows the response message in the generic error card instead of the sign-up card.

Firewall changes

The rule must exist before this deploys. checkRateLimit treats an unknown rule as "not rate limited", so without it signed-in free-model use is unlimited, and the gateway reports Firewall rate limit 'free-model-inference-user' is not configured to Sentry (source: free_model_inference_user).

1. Before merging: add the per-account rule

In Vercel, open the kilocode-ai-gateway project → Firewall → Configure → + New Rule, copy the settings of the existing "Free model rate limit" rule, and change the name, ID and limit:

Setting	Value
Name	Free model rate limit per account
If	Rate Limit API ID equals free-model-inference-user
Then	Rate Limit
Strategy	Fixed Window
Time window	3600 s
Request limit	300
Key	Header x-vercel-rate-limit-key
Action	Default (429)

Then Review Changes → Publish. After publishing, GET /v1/security/firewall/config/active?projectId=<kilocode-ai-gateway project id> should list a rule with condition rate_limit_api_id eq free-model-inference-user and rateLimit: { algo: "fixed_window", keys: ["header:x-vercel-rate-limit-key"], limit: 300, window: 3600 }.

2. After this is deployed: lower the anonymous per-IP limit

Edit the existing "Free model rate limit" rule (free-model-inference): change the request limit from 200 to 100, keeping the 3600 s window, then publish. Do this only after deploying: until then signed-in users still count against this per-IP rule.

3. Check after deploying

• Gateway logs show Model rate limit exceeded, user:<id>, … for signed-in callers and Model rate limit exceeded, ip:<ip>, … only for anonymous callers.
• Sentry has no Firewall rate limit 'free-model-inference-user' is not configured reports.

Verification

• NODE_ENV=test pnpm --filter web exec jest for llm-proxy.test.ts, free-model-rate-limit.test.ts, feature-detection.test.ts, and the bouncer suites: 255 passed (Node 24.14.1).
  • Anonymous callers count per IP, including server-side product requests from Cloudflare IPs, and get PROMOTION_MODEL_LIMIT_REACHED.
  • Signed-in callers count per account, with a bearer token (on their own IP and as a server-side product on a Cloudflare IP) and with a session cookie, and get the account message without the promotion code.
  • A rejected credential (malformed, or signature-valid but revoked) gets the invalid-token response before the limit check.
  • After the review fixes, the llm-proxy.test.ts and free-model-rate-limit.test.ts suites pass again: 137 tests.
  • Both limit responses carry x-should-retry: false.
  • Paid models and custom LLMs are still exempt.
• pnpm --filter @kilocode/web-shared typecheck: passed.
• oxlint and oxfmt on the changed files: clean.

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change (ef6c97d..08fac16) drops the token-claim fast path so the free-model rate-limit subject always comes from the resolved auth; the previous sign-up-prompt/returning-wrong-bucket warning is now fixed for all auth methods, and no new correctness, security, or memory-lifetime issue appears in the changed lines.

Assumptions: [INFERENCE] The per-account free-model-inference-user firewall rule (300/3600s) is published in Vercel before deploy; an unknown rule fails open, so the per-account cap does not exist until then (a manual pre-merge step the PR documents). [INFERENCE] Awaiting authPromise at the free-model check adds no latency because line 380 already awaits balanceAndSettingsPromise, which resolves only after auth. [INFERENCE] The removed isAnonymousUserId fast path only skipped a settled promise, so removed cost is negligible. [INFERENCE] Verification tests were not executed in this read-only review; the PR reports 137 passing tests in the affected suites. No memory-leak or unbounded-cache concern: the change introduces no new module state or per-request accumulation.

Files Reviewed (2 files)

• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts - 0 issues

Previous Review Summaries (2 snapshots, latest commit ef6c97d)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit ef6c97d)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change (4dfdb6d..ef6c97d) resolves the prior warning: free-model rate limiting now derives the subject from the resolved auth, so session-cookie callers count per account and a rejected credential returns the invalid-token response instead of the anonymous sign-up prompt.

Assumptions: [INFERENCE] The new free-model-inference-user firewall rule (300/3600s) is published in Vercel before deploy; checkRateLimit fails open for an unknown rule, so the per-account cap does not exist until then (a documented manual pre-merge step). [INFERENCE] A verified bearer token still short-circuits without a database query, and user.id from the session resolution is the same id the bearer path keys on, so both auth methods share one bucket. [INFERENCE] Verification tests were not executed in this read-only review; the PR reports 136 passing tests for the affected suites. [INFERENCE] The unrelated gateway-inference cap still keys session-cookie callers as anon:<ip> (pre-existing, outside the incremental diff). No memory-leak or unbounded-cache concern in the changed code: no new module state is introduced.

Files Reviewed (2 files)

• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts - 0 issues

Previous review (commit 4dfdb6d)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts	207	Signed-in callers who authenticate with the session cookie (no Authorization header) are keyed as anon:<ip>, stay on the per-IP free-model rule, and get the anonymous sign-up promotion response

Files Reviewed (8 files)

• packages/web-shared/src/lib/ai-gateway/free-model-rate-limit.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/free-model-rate-limit.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.ts - 1 issue
• packages/web-shared/src/lib/ai-gateway/handlers/llm-proxy.test.ts - 0 issues
• packages/web-shared/src/lib/ai-gateway/llm-proxy-helpers.ts - 0 issues
• packages/web-shared/src/lib/bouncer/inference.ts - 0 issues
• packages/web-shared/src/lib/feature-detection.ts - 0 issues
• packages/web-shared/src/lib/feature-detection.test.ts - 0 issues

Assumptions: [INFERENCE] The new free-model-inference-user firewall rule (limit 300, window 3600, header:x-vercel-rate-limit-key) is published in Vercel before this deploys; checkRateLimit fails open for an unknown rule, so signed-in free-model use is uncapped until it exists (the PR documents this as a manual pre-merge step). [INFERENCE] Server-side products (cloud-agent, code-review, app-builder, gastown) always send a signed account token, so removing the 401 leaves no anonymous Cloudflare-egress path that would share the per-IP bucket across users. [INFERENCE] Released Kilo clients handle error.code: "PROMOTION_MODEL_LIMIT_REACHED" and honor x-should-retry: false; that client code is in the kilocode repo and was not verifiable here. No memory-leak or unbounded-cache concerns in the changed code: the only new module state is the constant DO_NOT_RETRY_HEADERS object.

Fix these issues in Kilo Cloud

---

_{Reviewed by deepseek-flash-latest · Input: 45.5K · Output: 4.9K · Cached: 438.4K}

_{Review guidance: REVIEW.md from base branch main}