Skip to content

Dirty Runs: implement fenced A → B claims and controlled lease takeover #703

Open

Description

@jumski

Parent epic: #699. Depends on #702 and its reviewed contract #700.

Deliverable

Internal/versioned start, claim, and bounded controlled recovery functions for real A → B runtime tasks. Return the winning task and its claim_id; never treat message reservation or worker identity as claim ownership.

Start and admission

Validate the entire #700 envelope before runtime/message mutation. Create run, both step states, stable work row, A's task/message, and definition/protocol identity atomically. Queue A before start returns; B stays undispatched. Root A's existing input semantics and effective timeout apply. Work can be clean while A executes.

Match stored queue identity, mode, and exact step selector. Preserve guarded UPDATE ... RETURNING winners and actual PGMQ visibility checks from v1. Each proof claim/recovery call targets one explicit task/run and at most one message ID; reject larger or duplicate-filled arrays before expansion, scans, or mutation. There is no cross-run claim batch or recovery sweep in this slice. Use the shared lock order; only winning task rows reach handlers. Persist the owned physical message pair, lease/recovery horizon, and effective policy in history.

New claims require a started run, eligible started step/queued task, compatible identity, and clean engine health/caught-up generations. Dirty/retry-wait/attention tasks cannot newly execute. Reject stale/wrong-message/wrong-route inputs without archive or visibility writes from the claim function. PGMQ reservation itself remains a separate earlier read.

Controlled lease recovery/takeover

Use the protocol decision from #700 for horizon expiry versus locked revocation. Candidate selection is not authority. Under run → work → ordered step/task → queue locks, recheck eligibility/horizon/current token, revoke the old claim, count one infrastructure loss, and requeue or create the permitted replacement claim atomically. Never change handler-failure accounting to manufacture exhaustion.

Preserve the old history and apply visibility only to the task's actual stored pair. Bounded history/one controlled takeover is an internal proof limit, not #689's automatic worker-loss policy. On reaching that limit, durable engine health blocks further claims without archiving proof messages or pretending business exhaustion succeeded.

Do not add a cron recovery driver or strict worker-level ownership (#697). Multiple competing workers are allowed; only one current task claim wins.

Acceptance

Add focused pgTAP and deterministic separate-session barriers. No random sleep or sequential simulation counts as proof of the race.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpkgs/corepriority:p1Next batch: current correctness, user blocker, or operational safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions