Skip to content

fix(auto-triage): serialize pending ticket dispatch per owner - #7906

Closed
iscekic wants to merge 6 commits into
mainfrom
drain/harden-triage-dispatch-serialization
Closed

iscekic wants to merge 6 commits into
mainfrom
drain/harden-triage-dispatch-serialization

Conversation

@iscekic

@iscekic iscekic commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Change

Pending auto-triage ticket dispatch now claims and dispatches tickets one at a time:

  • claimNextPendingTicket reads the owner's active count, selects the oldest pending ticket, and moves it to 'analyzing' inside one transaction serialized by a per-owner advisory lock (pg_advisory_xact_lock, namespaced with a feature key as other owner-scoped paths do). Every claim re-reads the count, so the per-owner ceiling stays authoritative across overlapping triggers.
  • The claim is a status-conditional UPDATE that only applies while the row is still 'pending' and returns the claimed row; dispatchTicket runs only for the ticket just claimed.
  • When dispatch fails after a claim, the ticket is returned to 'pending' with its start time cleared so a later pass can retry it, and that ticket is skipped for the rest of the current pass. Failures are logged and captured in Sentry; no error message is persisted, so a recovered ticket renders without a stale dispatch error.
  • updateTriageTicketStatus accepts an optional expectedStatus that restricts the update to rows still in that state.

Verification

  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.test.ts (7 tests, DB-backed against the migrated per-worker test database). Concurrent calls process disjoint ticket sets covering all pending tickets, dispatch volume stays within the per-owner limit, no slots are used when the owner is at the limit, each dispatch observes at most one claimed ticket, a failed dispatch returns the ticket to 'pending' with no error message and a later pass dispatches it, and the optional expected-status restriction is exercised. All pass:
    cd apps/web && NODE_ENV=test npx jest src/lib/auto-triage/ — 3 suites, 12 tests, green.
  • pnpm --filter web run typecheck (trpc build + tsgo): exit 0.
  • pnpm --filter web run lint: 0 warnings, 0 errors.
  • pnpm -w exec oxfmt on the touched files: no differences.

Proxy answers

  1. Question: How should the pending ticket claim be made atomic?
    Answer: Claim + lock (Recommended)
    Confidence: 0.90 · Precedent: Fix the root cause and the class of the issue: 'fix class of issue preventing progress' (7f3470e0ba6f). Only lock plus claim closes both the duplicate flip and the cap overrun, and it matches the brief and the sibling pattern.
  2. Question: When dispatchTicket throws after a successful claim (e.g. worker unreachable), what should happen to the ticket?
    Answer: Reset to pending on failure
    Confidence: 0.70 · Precedent: Never accept 'blocked' as an end state: 'a work item must converge' (2f71af3a787e). The brief requires a failed claim to go back to 'pending' or to the existing failure status. A ticket stranded in 'analyzing' never converges. Use the existing failure status where that path already applies, and reset to pending otherwise.
  3. Question: Should updateTriageTicketStatus gain the optional expected-status guard even though the dispatch path will no longer use it for the claim?
    Answer: Yes, optional expectedStatus (Recommended)
    Confidence: 0.75 · Precedent: Fix the class, not the single symptom: 'if anything repeats the same mistake over and over again, fix the class' (00eb2a1d3e4b). The brief asks for the guard explicitly, mirroring interruptTriageTicket.

@iscekic iscekic added drain Work item from the findings drain janitor:trust Janitor finding: trust boundary hardening labels Oct 17, 2026
@kilo-code-bot

kilo-code-bot Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental rewrite claims and dispatches one pending ticket at a time under the per-owner advisory lock, and the previously flagged startedAt/stale-error findings are resolved; no new issues found.

Files Reviewed (2 files)
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts - claim-one-at-a-time loop; status-conditional claim under pg_advisory_xact_lock, slot ceiling and pending count re-read every iteration, failed tickets excluded via notInArray and returned to pending without persisting an error message
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.test.ts - new tests for single-reservation behavior and error-message-free recovery
Previous Review Summaries (4 snapshots, latest commit ad204d0)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit ad204d0)

Status: No Issues Found | Recommendation: Merge

Executive Summary

The incremental change widens UpdateTicketParams.startedAt to Date | null, resolving the previously flagged type errors at dispatch-pending-tickets.ts:164 and triage-tickets.ts:171; no new issues found.

Files Reviewed (1 file)
  • apps/web/src/lib/auto-triage/db/types.ts - widened startedAt to Date | null; the startedAt: null reset now type-checks under strict, and the redundant intersection in triage-tickets.ts is now valid

Previous review (commit 4a82abf)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The attempted fix for the startedAt: null type error is ineffective: adding startedAt?: Date | null inside the intersection does not change the property type, because UpdateTicketParams already declares startedAt?: Date, so the strict typecheck still fails.

Overview

Severity Count
CRITICAL 1
WARNING 0
SUGGESTION 0
Issue Details (click to expand)

CRITICAL

File Line Issue
apps/web/src/lib/auto-triage/db/triage-tickets.ts 171 UpdateTicketParams & { startedAt?: Date | null } does not widen startedAt (intersection narrows the property to Date), so the startedAt: null reset at dispatch-pending-tickets.ts:164 still fails the strict typecheck. Widen UpdateTicketParams.startedAt itself, or use Omit<UpdateTicketParams, 'startedAt'> & { startedAt?: Date | null; expectedStatus?: TriageStatus }.
Files Reviewed (1 incremental file)
  • apps/web/src/lib/auto-triage/db/triage-tickets.ts - 1 issue (widened param is ineffective)
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts - re-verified; the startedAt: null reset still does not compile

Fix these issues in Kilo Cloud

Previous review (commit f476574)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The stale-started_at reset from the previous review is fixed, but the new reset passes startedAt: null against a param typed Date, which fails the strict typecheck (typecheck (web) and build are red on this commit).

Overview

Severity Count
CRITICAL 1
WARNING 0
SUGGESTION 0
Issue Details (click to expand)

CRITICAL

File Line Issue
apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts 164 startedAt: null is not assignable to UpdateTicketParams.startedAt?: Date under strict: true; widen the param to Date | null.
Files Reviewed (2 files)
  • apps/web/src/lib/auto-triage/db/triage-tickets.ts - no issues (the startedAt ? … : null branch correctly clears the timestamp, but it depends on the param being widened)
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts - 1 issue

Fix these issues in Kilo Cloud

Previous review (commit d85eda5)

Status: 1 Issue Found | Recommendation: Address before merge

Executive Summary

The per-owner advisory lock plus status-conditional claim is sound; the only finding is that a failed dispatch returns the ticket to pending while leaving its started_at set.

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 1
Issue Details (click to expand)

SUGGESTION

File Line Issue
apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts 161 Reset to pending after a failed dispatch does not clear started_at, so a queued ticket renders as "Started …" in the auto-triage UI.
Files Reviewed (3 files)
  • apps/web/src/lib/auto-triage/db/triage-tickets.ts - no new issues (added expectedStatus guard is correct)
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.ts - 1 issue
  • apps/web/src/lib/auto-triage/dispatch/dispatch-pending-tickets.test.ts - no issues

Fix these issues in Kilo Cloud


Reviewed by deepseek-v4.1-flash · Input: 54.9K · Output: 9.7K · Cached: 536.8K

Review guidance: REVIEW.md from base branch main

@iscekic

iscekic commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

bot: Fixed: the failed-dispatch reset now clears started_at (and the helper accepts a null timestamp), so a ticket returned to 'pending' no longer renders as started. Tests updated and green.

@Kilo-Org Kilo-Org deleted a comment from kilo-code-bot Bot Oct 17, 2026
Each pending ticket is claimed in its own lock-serialized transaction
immediately before its dispatch call, so the active-ticket budget is
re-read on every claim and a pass only ever holds the reservation it is
dispatching. A ticket returned to 'pending' after a dispatch error
carries no persisted error message, so a later successful dispatch
renders clean.
@iscekic iscekic added the merge:human Substantive functional risk requires human review. label Oct 19, 2026
@iscekic
iscekic enabled auto-merge October 19, 2026 17:47
@iscekic

iscekic commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

bot: The auto-triage subsystem this branch changes was removed from main before this session started: #8151 removed the service, web library, routers, and webhook handlers, and #8152 dropped the related table. Every file this branch touches is deleted on main, so there is no remaining code path to modify and the change has no target. Closing as obsolete.

@iscekic iscekic closed this Oct 19, 2026
auto-merge was automatically disabled October 19, 2026 21:47

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

drain Work item from the findings drain janitor:trust Janitor finding: trust boundary hardening merge:human Substantive functional risk requires human review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant