Skip to content

fix(cloud-agent-sdk): keep the live stream when a snapshot read fails - #8539

Merged
iscekic merged 4 commits into
mainfrom
drain/cloud-agent-transport-snapshot-read-retry
Oct 22, 2026
Merged

iscekic merged 4 commits into
mainfrom
drain/cloud-agent-transport-snapshot-read-retry

Conversation

@iscekic

@iscekic iscekic commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Evidence

Evidence: demonstrated — the initial session exercised forced snapshot failures on the local Android app and observed continued live delivery. After merging main, a fresh loopback HTTP/WebSocket smoke exercised the merged SDK's initial-read and reconnect recovery paths.

  • Initial Android proof: session open, live delivery during failed reads, and history recovery. This follow-up did not rerun Android; it reconciles the transport with main's shared snapshot helpers.
  • Merged-tree SDK suite: pnpm --filter @kilocode/cloud-agent-sdk test --runInBand — 35 suites, 1,790 tests passed. Jest printed its delayed-exit warning and the command exited successfully.
  • pnpm --filter @kilocode/cloud-agent-sdk typecheck, formatting, targeted oxlint, and git diff --check passed. pnpm install --frozen-lockfile passed.
  • Fresh runtime smoke: real loopback HTTP reads and WebSocket connections, real SDK transport and normalizer, fixture-only data. The initial paginated read returned HTTP 503; the socket delivered a live message and the next read restored history. Reconnect catch-up returned HTTP 503 on each of the paginated and legacy seams; each reported once, retried, restored history, and delivered a live message. All three runs recorded the initial page once and retained the completed tool's event time. Read-attempt offsets were 21/527 ms (initial), 3/668/1169 ms (paginated reconnect), and 1/1050/1551 ms (legacy reconnect).

Change

  • Snapshot-read rejections do not discard an available stream ticket. The live socket remains useful while history recovery runs.
  • Rejected reads and typed retryable_failure outcomes report once per failure episode and retry three times at 500 ms, 1 s, and 2 s. Missing sessions and terminal typed outcomes stop retries. Disconnect, destroy, and lifecycle changes cancel pending work.
  • The first successful page per connect seeds the manager. Later catch-up or in-flight retry pages replay without resetting the older-history cursor. A late page does not regress a cursor already advanced by live events.
  • Terminal typed history failures still report on initial open, not on every reconnect.
  • The main-merge resolution retains main's shared replaySnapshotPage, snapshotPageFromSnapshot, and sessionSnapshotFailureMessage helpers, removing the transport-local duplicates.

Coverage and limitations

Eight recovery tests cover initial rejection, bounded retries, both reconnect read seams, cursor non-regression, the initial/reconnect race, retryable typed failure, and terminal reconnect outcomes. The existing suite also covers replayed message parts and settled tool timing.

The original Android run used the real local web tRPC route, stream worker, and session ingest with deterministic inference. Its temporary read-failure patch was removed before commit. The fresh follow-up smoke uses a loopback fixture server, not the full application stack. No CLI relay protocol changed; this is the app-facing SDK transport.

Proxy answers

The owner's proxy gave these answers in the owner's place.

  1. Question: Bounded retry budget for a failed snapshot read? Answer: Three retries, 500 ms base, 4 s cap, about 3.5 s total. With three retries, the actual delays are 500/1000/2000 ms; the unreachable cap constant was removed after review. Confidence: 0.60. Precedent: Prefer the least work without reducing quality; cover transient failures deterministically.
  2. Question: Error callback frequency during snapshot recovery? Answer: Once per failure episode. Confidence: 0.65. Precedent: Preserve clear UX without repeatedly flashing the same banner.
  3. Question: Retry typed retryable_failure reconnect outcomes? Answer: Yes, with the same bounded backoff and one report. Confidence: 0.85. Precedent: Fix the class of failure, not only one symptom.
  4. Question: How should the Android run force a snapshot failure? Answer: Use a temporary, uncommitted patch to the web tRPC getSessionMessagesPage route. Confidence: 0.70. Precedent: Authorized local E2E fault injection; remove the patch before committing.
  5. Question: Retry typed retryable_failure outcomes on the initial read too? Answer: Yes, with the same bounded backoff. Confidence: 0.85. Precedent: Apply the same recovery rule to the whole defect class and satisfy the initial-read requirement.

The initial connect awaited the stream ticket and the snapshot page in one
`Promise.all`, so a rejected page read discarded an already-available ticket
and no websocket opened. The reconnect catch-up reads attached no-op
rejection handlers, so a failed catch-up left a transcript gap with no error,
no retry and no signal.

The ticket and the snapshot read now start together and only a failed ticket
stops the connect. A failed read is reported once per episode through the
transport error callback and retried with bounded exponential backoff (three
retries, 500 ms base, 4 s cap) while the socket keeps serving live events. A
retried page replays when it lands, records the initial page exactly once, and
never regresses a replay cursor that live events already advanced. The initial
bounded read also retries a typed `retryable_failure`, so a transient history
read no longer leaves the transcript empty until the next reconnect.
@iscekic iscekic added drain Work item from the findings drain janitor:performance-reliability Janitor performance-reliability findings labels Oct 22, 2026
@kilo-code-bot

kilo-code-bot Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The follow-up commit keeps terminal typed history failures off the reconnect error path while still reporting them once on the initial open, and the added test covers the reconnect catch-up case.

Assumptions: [INFERENCE] Tests were not executed in this read-only review; behavior was verified by reading cloud-agent-transport.ts, the new test, and the base-branch reconnect path.

Files Reviewed (2 files)
  • packages/cloud-agent-sdk/src/cloud-agent-transport.ts
  • packages/cloud-agent-sdk/src/cloud-agent-transport.test.ts
Previous Review Summaries (2 snapshots, latest commit 724669e)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 724669e)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The follow-up commit closes both earlier findings: replayInitialPage now owns the single-page-per-connect guard so the initial retry chain and reconnect catch-up share one decision, and the unreachable delay cap constant is removed; the new race test covers a reconnect catch-up landing before a late initial retry.

Assumptions: [INFERENCE] Tests were not executed in this read-only review; the race behavior and the guard were verified by reading cloud-agent-transport.ts and the new test.

Files Reviewed (2 files)
  • packages/cloud-agent-sdk/src/cloud-agent-transport.ts
  • packages/cloud-agent-sdk/src/cloud-agent-transport.test.ts

Notes: the guard reset in connect() keeps one initial page per connect; disconnect()/destroy() plus cancelSnapshotRetry() still clear the bounded retry timer, so no new timer leak. Removing SNAPSHOT_READ_MAX_DELAY_MS leaves no dangling references, and the delays remain 500 ms / 1 s / 2 s as documented.

Previous review (commit a28ce8d)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The snapshot-read degradation fix is sound and well tested, but the initial retry chain reuses replayInitialPage without the initialPageRecorded guard used by the reconnect path, so a superseded in-flight retry can re-fire onInitialPageLoaded and reset the older-pages cursor.

Assumptions: [INFERENCE] Tests were not executed in this read-only review; conclusions about the retry race rest on reading base-connection.ts reconnect timing and the manager's recordInitialPage. The legacy fetchSnapshot seam and mobile manager behavior were reviewed by reading only.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 2
Issue Details (click to expand)

SUGGESTION

File Line Issue
packages/cloud-agent-sdk/src/cloud-agent-transport.ts 318 Initial retry chain replays through replayInitialPage, bypassing the initialPageRecorded guard; cancelSnapshotRetry() cannot cancel an in-flight read, so a late retry can call onInitialPageLoaded a second time and reset the older-pages cursor.
packages/cloud-agent-sdk/src/cloud-agent-transport.ts 296 SNAPSHOT_READ_MAX_DELAY_MS (4000 ms) is unreachable with 3 retries (delays 500/1000/2000 ms), so the documented 4 s cap never applies.
Files Reviewed (2 files)
  • packages/cloud-agent-sdk/src/cloud-agent-transport.ts - 2 issues
  • packages/cloud-agent-sdk/src/cloud-agent-transport.test.ts - no issues (new 6-test recovery block covers initial rejection, retry budget, reconnect on both seams, cursor non-regression, and typed retryable failure)

Notes: no new retry timer leak found — cancelSnapshotRetry() clears the pending timer on connect(), disconnect(), and destroy(), and the retry chain is bounded at 3. The typed-failure message refactor preserves the previous text exactly.

Fix these issues in Kilo Cloud


Reviewed by deepseek-flash-latest · Input: 54.1K · Output: 8.5K · Cached: 556.2K

Review guidance: REVIEW.md from base branch main

@iscekic

iscekic commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

bot: Android E2E on the local dev stack (nextjs + Metro + cloud-agent-next worker + deterministic fake model), emulator omp36, owned fake account. One Cloud Agent session created through the real web tRPC surface; the web page-read route was made to fail on demand for the run (temporary local patch, reverted before the commit).

  1. 01-read-failed.png — opening the session while its snapshot read failed: 4 page-read attempts were logged at t+0.00 s, +0.86 s, +1.94 s, +3.99 s (initial read plus three bounded retries, then stop), the read error reached the session error state, and the websocket still opened.
  2. 02-live-events.png — same session, same failing read, a follow-up turn sent from outside the app: the user message and the assistant reply arrived and rendered live, and the error state cleared.
  3. 03-history-restored.png — the fault removed, reopened: the full transcript loads from the snapshot page, including the exchange that was produced before the session was first opened and therefore never delivered over the socket.

Each frame was captured in the same command as its uiautomator dump, so the on-screen assertion is recorded alongside the image. No private data: owned fake account, local stack, deterministic model.

Session open with the snapshot read failing

Live turn received while the snapshot read failed

Transcript loaded from the snapshot after the fault cleared

…e dead retry cap

A reconnect catch-up read can land while an initial retry is still in flight,
and that retry then handed the manager a second page for the same connect,
resetting the older-pages cursor back to the newest page. `replayInitialPage`
now records the page only when nothing has seeded the session yet, so the
first page wins whichever chain delivers it, and the reconnect path no longer
needs its own copy of that decision.

The 4 s delay cap was unreachable with three retries (500 ms, 1 s, 2 s), so
the delay is the plain bounded exponential and the constant is gone. Adds a
test where a reconnect catch-up wins the race against a late initial retry.
…t path

`applySnapshotRead` reported every non-page outcome, so a reconnect catch-up
that came back `too_large` or `invalid_data` raised the session error on every
reconnect for history that can never be loaded. Reporting now covers only the
transient outcomes the retry exists for; the terminal typed failure is still
surfaced once by the initial read, exactly where it was before.

Adds a test where a reconnect catch-up returns `too_large` and must not reach
the error channel.
@iscekic iscekic added the merge:bot Bot-owned review; merge when ready once readiness passes and a review approves. label Oct 22, 2026
…nsport-snapshot-read-retry

# Conflicts:
#	packages/cloud-agent-sdk/src/cloud-agent-transport.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

drain Work item from the findings drain janitor:performance-reliability Janitor performance-reliability findings merge:bot Bot-owned review; merge when ready once readiness passes and a review approves.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants