Skip to content

fix(gateway): preserve retry decisions and catalog errors - #15367

Merged
marius-kilocode merged 5 commits into
v2from
gateway-error-parity
Oct 22, 2026
Merged

marius-kilocode merged 5 commits into
v2from
gateway-error-parity

Conversation

@marius-kilocode

@marius-kilocode marius-kilocode commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What Problem This Solves

Permanent Gateway errors in HTTP-200 SSE streams lost their retry decision before native classification. Paid-auth and promotion failures could repeat instead of stopping. Catalog failures also lost their cause before a selected Kilo model reached the CLI error boundary.

Why This Change Was Made

The initial comparison used main 9e4e3a43c802 and v2 981607bf544. Main was refreshed to 4481e06d8af1; the four relevant retry/error files are unchanged. Current v2 f50d908f132, including merged #15366, is integrated normally. No main merge or open peer-branch merge is included.

Main has dedicated terminal handling for PAID_MODEL_AUTH_REQUIRED and PROMOTION_MODEL_LIMIT_REACHED. Its stream parser supports numeric/string codes, JSON-wrapped errors and response.failed. Billing documentation uses numeric 402 with error.metadata.buyCreditsUrl.

Review exposed a main behavior that should not be copied: its broad prose fallback can retry deterministic errors mentioning 500 tokens or Provider returned error, and HTTP 429 exhausted quota can retry through the SDK flag. This PR deliberately retains v2's native permanent classifications instead. Unknown provider failures also keep the native retry default. Only known Kilo terminal signals and the existing top-up exception override it; no new error framework or retry engine is added.

User Impact

  • Preserve numeric/string codes, useful structured details, billing links, and actionable HTTP/SSE messages. A JSON string in message cannot replace a structured outer error.
  • Apply Kilo-only normalization to stream, buffered generate, and native trigger compact. Generic validation, quota, rate-limit, server, unknown, and non-Kilo provider behavior stays with the native classifier.
  • Known Kilo terminal errors remain terminal even with a positive retry hint. Other explicit retry headers, Retry-After, bounded retry/backoff, and interruption behavior remain intact.
  • Return schema-validated catalog failure data (kind, optional HTTP status) through the existing models RPC. The host uses a shared safe formatter, not message parsing. No raw response, credential, endpoint query, or organization ID is included.
  • Missing membership after a successful profile check reports an unavailable team and selection guidance, not a synthetic HTTP 403 or invalid credentials. Personal last-good catalogs and team fail-closed/retry behavior are preserved.

Review Decisions

Finding Change
Outer JSON message discarded a terminal code Structured outer envelopes take precedence; HTTP/SSE regressions added
Prose/positive headers overrode native permanent errors; generic api_error bypassed its guard Removed prose retry heuristics and generic positive overrides; native quota/validation/unknown decisions retained; explicit free cap retained
compact bypassed normalization Same wrapper applied to native trigger compaction, with counted permanent/top-up regressions for both SSE shapes
Bare frame marker became error.type One error serializer uses the nested error type or original code, not the envelope marker
Catalog regex duplicated message grammar Typed RPC data and one shared safe formatter; arbitrary RPC text is ignored
Missing team looked like HTTP/credential failure Added organization_missing with selection guidance and no fabricated status
Runtime schema import remained a devDependency Moved @opencode/schema to runtime dependencies; no version changes
Unread failure on healthy last-good state Removed the unused assignment
Cache/coalescing or a separate catalog-status RPC Not added: the current RPC is local and runs only after resolution fails, not on the healthy per-step path; another cache/API is not justified

All six addressed remote review threads were resolved after the fixes were pushed, without replies.

Evidence

Bun 1.4.2. Package tests used fresh OPENCODE_TEST_HOME and XDG directories. Integrated-branch checks:

Command Result
bun test in packages/kilo-gateway-core 78 passed, including merged usage metadata tests
bun test test/catalog-errors.test.ts test/gateway.test.ts in packages/kilo-plugin 21 passed
bun test test/gateway-errors.test.ts test/catalog-errors.test.ts test/host.test.ts test/rpc.test.ts test/gateway-usage.test.ts in packages/kilo-cli 29 passed, 448 assertions; includes 26 error/host/RPC tests plus merged usage integration
bun test test/kilocode/gateway-usage.test.ts test/kilocode/session-stats.test.ts in packages/core 17 passed
bun test upstream/progress.test.ts upstream/placement.test.ts in script 18 passed
Affected package typechecks and root bun run check Passed, 28/28 root package checks
bun run generate in packages/client No tracked API drift
Progress regeneration twice, strict --check, marker/shared-patch/import checks, git diff --check Passed

Tests execute production normalization, native OpenRouter/OpenAI Responses/Anthropic parsing, toSessionError, and SessionRunnerRetry.policy. Counted buffered/native-trigger retries use ModelRetry.retry and TestClock: paid-auth/promotion stop after one request; top-up recovers on request two, for both error and response.failed. Trigger compaction returns the real encrypted checkpoint. This is not a claim that the JSON compact endpoint accepts SSE.

The nested worktree initially selected ancestor v1 Node types. bun install --linker hoisted --frozen-lockfile corrected generated dependency layout only. No source workaround, version change, type relaxation, or disabled check was added.

Real CLI And TUI Self-Tests

Used controlled counted loopback fixtures, fresh profiles, --standalone, and an allowlisted env -i child environment. No production balance, entitlement, policy, or credential was changed. Fixture/harness processes were stopped.

CLI command: bun dev run --standalone --model kilo/fixture/<case> <prompt>, with a fresh KILO_V2_PROFILE/HOME, fixture KILO_API_URL, pinned Bun PATH, TERM, and KILO_DISABLE_MODELS_FETCH=true.

Actual flow Primary requests Result
HTTP/SSE paid-auth and promotion 1 each Terminal sign-in/signup guidance
Responses and Anthropic auth envelopes 1 each Terminal sign-in guidance
HTTP/SSE promotion with unrelated JSON in outer message 1 each Outer terminal code retained
HTTP 400, 500 tokens or provider-error prose, including message-only body 1 each Native validation decision stops
HTTP 429 insufficient_quota 1 Exhausted quota stops, not a throttle retry
HTTP 402 balance error with credit URL 1 Balance guidance and https://app.kilo.ai/credits displayed
Transient HTTP 503/SSE server error, twice then success 3 each Fixture reply returned
SSE top-up with generic invalid-request type, twice then success 3 Explicit Kilo transient exception retained
Cold catalog HTTP 503, malformed response, or network failure with explicit model 0 generation Safe cause plus connection/credential guidance
Missing team after valid fixture profile 0 generation Select/clear-team guidance, no fake HTTP 403

After integrating current v2, repeated HTTP/SSE promotion, exhausted quota, credits, top-up, native transient, and an ordinary successful CLI call. Primary failures stayed at one request; transients recovered at three. The merged client integration also verified routed costs, auxiliary calls, statistics, and persistence across a server restart.

The vscode-self-test session gateway-errors-e-20261022 used --mode none and isolated workspace/profile. Real TUI picker selection and submission produced one request each for auth/promotion and terminal guidance; transient recovery used three requests and settled in 5.7 seconds. Explicit-model catalog guidance was also read in the harness terminal. Not every policy code or native compaction was manually driven through the TUI; those have actual-path automated coverage.

Placement

Change Rung Why not lower Proof
Wire errors and catalog RPC 1 Existing response hook and Gateway RPC cover these boundaries kilo-gateway-core/src/errors.ts, error branch of stream.ts, plugin and schema Gateway contract
Retry normalization 2 Native execution bypasses AISDK hooks; a retry hook alone misses other LLM entry points kilo-cli/src/gateway-errors.ts, existing LLMClient node decorator
Selected-model diagnostic 2 Unavailability fails before provider/model hooks kilo-cli/src/gateway-model.ts, existing runner node decorated with scoped RPC

Relative to current v2, no shared upstream source, generic classifier default, or new mirror is changed. Logic stays in Kilo-owned paths. Three new core-internal imports provide the existing LLM node, scoped RPC, and runner boundary (effect/app-node-platform, rpc, session/runner/model). Runtime workspace manifest/lock changes are required by those host consumers. The progress tool fix below is Kilo-owned tooling.

Progress map

Current base gateway: 61/75 to 61/75, 12 dropped; module 31/35, patch 12/18, test 18/22. The initial 59/75 became 61/75 through already-merged #15366, not extra claims in this error PR. Two existing error proofs now cover actual streamed/buffered/native-compaction execution instead of parser-only evidence. No error cell is newly closed or dropped.

Core-internal imports grow from 9 to 12 for the host boundaries above. Shared patches remain at the current base's 36; this PR adds none. The 14 existing open Gateway cells (cloud/event clients, remaining handlers/request transforms, and associated tests) remain outside this error change.

CI Follow-Up

On prior head 5436a8607f8, CI's synthetic merge followed v2's first parent while local regeneration followed feature-branch history. That made only README.md and history.svg stale. 66570fd2c08 anchors history to origin/v2, adds a regression, and regenerates those artifacts with unchanged cell totals and strict checks.

The same run failed Git trees > refresh handles check-ignore exit 128 with a Linux EPIPE. Git source/test, process/spawner and fixtures are byte-identical to current v2 and pinned upstream. Isolated macOS checks passed the related cases 75 times, exit 128 another 100 times, the full Git file, and 40 production-path probes. This does not prove the Linux issue is fixed. No swallowing, skip, retry addition, workflow rerun, or unrelated Git change was used. On head 66570fd, all GitHub Actions checks passed, including the previously failing Linux unit and progress-map jobs. The external Kilo Code Review check remained a terminal failure; it is not reported green. Human review was approved, there were no unresolved GitHub review threads, and the PR was merged at 10:34 UTC as e346cbf.

Limits

  • A cold TUI can fall back to an available upstream model when the configured Kilo catalog is unavailable, before a selected-Kilo run reaches the diagnostic. Initial-picker guidance is not claimed. Explicit-model CLI guidance is verified; default/picker policy and successful empty catalogs are unchanged.
  • No deployed paid failure was induced. Billing/policy production behavior beyond the inspected main contract and fixture tests is not claimed.
  • Generic retry decisions intentionally use v2 native behavior rather than main's faulty broad prose fallback.

Manual Test

  1. Inject auth/promotion errors in HTTP and HTTP-200 SSE, including an unrelated JSON outer message. Submit once and confirm one request plus guidance.
  2. Inject HTTP 400 validation and HTTP 429 exhausted-quota errors. Confirm no retry. Inject HTTP 402 with credit metadata and confirm the purchase guidance.
  3. Return two server/top-up errors then success. Confirm three requests and the reply.
  4. Use an explicit Kilo model with a cold catalog failure or missing team. Confirm safe actionable guidance and no generation request.

@github-actions github-actions Bot added the area:gateway Kilo Gateway and auth label Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

➖ v2 migration progress: 0.9% → 0.9% (0 done, 0 total, 0 dropped; 92/9829 cells)

Placement: 0 new marked lines in shared files, 1 new kilo_port annotation, 3 new core-internal imports (@opencode/core/effect/app-node-platform, @opencode/core/rpc, @opencode/core/session/runner/model).

Ratchets: 12 core-internal imports (base 9); shared patches by rung 20 seam, 16 patch (base 20 seam, 16 patch).

No change to the progress map.

v2 migration progress map at this commit

Full map · how cells close

Comment thread packages/kilo-cli/src/gateway-model.ts Resolved
Comment thread packages/kilo-cli/src/gateway-model.ts Resolved
Comment thread packages/kilo-cli/package.json Resolved
@kilo-code-bot

kilo-code-bot Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 4 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 3
Issue Details (click to expand)

WARNING

File Line Issue
packages/kilo-gateway-core/src/errors.ts 46 The unconditional JSON unwrap of a top-level message can discard a valid outer error envelope, so a terminal Kilo code (e.g. PROMOTION_MODEL_LIMIT_REACHED on 429) parses as no error, no x-should-retry: false is emitted, and the cap is retried.

SUGGESTION

File Line Issue
packages/kilo-gateway-core/src/stream.ts 91 For envelope-less {type:"error", code, message} frames, parsed.details is the frame itself, so error.type becomes "error" and Anthropic renders error: <message> instead of the provider code.
packages/kilo-plugin/src/gateway/index.ts 245 A missing team membership is synthesized as HTTP 403 and reported as "Check your connection and credentials", but no HTTP request failed and the credentials are valid.
packages/kilo-cli/package.json 22 gateway-model.ts now imports @opencode/schema as a runtime value (the first src import of it in this package), but it remains a devDependency, unlike @kilocode/gateway-core which this PR moved to dependencies.
Files Reviewed (18 files)
  • bun.lock
  • migration-tracking/progress/README.md
  • migration-tracking/progress/dispositions.json
  • migration-tracking/progress/history.svg
  • migration-tracking/progress/progress.json
  • packages/kilo-cli/package.json - 1 suggestion
  • packages/kilo-cli/src/gateway-errors.ts
  • packages/kilo-cli/src/gateway-model.ts
  • packages/kilo-cli/src/host.ts
  • packages/kilo-cli/test/catalog-errors.test.ts
  • packages/kilo-cli/test/gateway-errors.test.ts
  • packages/kilo-cli/test/rpc.test.ts
  • packages/kilo-gateway-core/src/errors.ts - 1 warning
  • packages/kilo-gateway-core/src/stream.ts - 1 suggestion
  • packages/kilo-gateway-core/test/requests-errors.test.ts
  • packages/kilo-plugin/src/gateway/index.ts - 1 suggestion
  • packages/kilo-plugin/test/catalog-errors.test.ts
  • packages/schema/src/kilocode/gateway.ts

The follow-up commit 317fecd replaces the CLI's duplicate message regex with typed CatalogFailure RPC data and adds the schema/test coverage for it; that previous finding is resolved. No new issues were found in the incremental diff, and no memory leaks were introduced: the failure data is a plain value, the layer diagnostic effect is still evaluated lazily, and the catalog state lifecycle is unchanged.

Previous Review Summary (commit 974f78c)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 974f78c)

Status: 5 Issues Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 4
Issue Details (click to expand)

WARNING

File Line Issue
packages/kilo-gateway-core/src/errors.ts 40 The unconditional JSON unwrap of a top-level message can discard a valid outer error envelope, so a terminal Kilo code (e.g. PROMOTION_MODEL_LIMIT_REACHED on 429) parses as no error, no x-should-retry: false is emitted, and the cap is retried.

SUGGESTION

File Line Issue
packages/kilo-gateway-core/src/stream.ts 91 For envelope-less {type:"error", code, message} frames, parsed.details is the frame itself, so error.type becomes "error" and Anthropic renders error: <message> instead of the provider code.
packages/kilo-plugin/src/gateway/index.ts 245 A missing team membership is synthesized as HTTP 403 and reported as "Check your connection and credentials", but no HTTP request failed and the credentials are valid.
packages/kilo-cli/src/gateway-model.ts 52 The CLI regex re-implements the plugin's diagnostic grammar; a status-less unauthorized/http failure allowed by KiloModels.Failure would silently drop the diagnostic.
packages/kilo-cli/package.json 22 gateway-model.ts now imports @opencode/schema at runtime, but it remains a devDependency.
Files Reviewed (16 files)
  • bun.lock
  • migration-tracking/progress/README.md
  • migration-tracking/progress/dispositions.json
  • migration-tracking/progress/history.svg
  • migration-tracking/progress/progress.json
  • packages/kilo-cli/package.json - 1 suggestion
  • packages/kilo-cli/src/gateway-errors.ts
  • packages/kilo-cli/src/gateway-model.ts - 1 suggestion
  • packages/kilo-cli/src/host.ts
  • packages/kilo-cli/test/catalog-errors.test.ts
  • packages/kilo-cli/test/gateway-errors.test.ts
  • packages/kilo-gateway-core/src/errors.ts - 1 warning
  • packages/kilo-gateway-core/src/stream.ts - 1 suggestion
  • packages/kilo-gateway-core/test/requests-errors.test.ts
  • packages/kilo-plugin/src/gateway/index.ts - 1 suggestion
  • packages/kilo-plugin/test/catalog-errors.test.ts

No CRITICAL findings: the model/error wrapper, LayerNode replacement, and plugin catalog state were checked for recursion, identity, and lifecycle problems and are sound. No memory leaks were introduced. Retry decisions match v1 (origin/main) except where noted above.

Fix these issues in Kilo Cloud


Reviewed by deepseek-flash-latest · Input: 203.1K · Output: 26.3K · Cached: 955.5K

Review guidance: REVIEW.md from base branch v2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:gateway Kilo Gateway and auth

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants