[Thom Chiovoloni (thomcc-work)]

Summary

Record completed_at for ContinuedAsNew executions so age-based retention can remove old generations of long-running workflows. Previously, those timestamps remained NULL, leaving continued executions ineligible for age-based pruning.

This also adds a paginated SQL pruning API and coordinates locking across pruning and deletion operations to protect live executions during concurrent maintenance.

Changes

Completion timestamps — migration 0027

• Set completion timestamps when an execution continues as new.
• Backfill missing timestamps from the last persisted OrchestrationContinuedAsNew history event, using its original timestamp rather than the upgrade time.
• Preserve existing timestamps and other execution statuses. Rows without a recoverable terminal event remain NULL and produce a warning.
• Retain the acknowledgement lease safeguards introduced in fix: make orchestration lock renewal and acknowledge safe under concurrency #44.
• Fail closed on malformed history JSONB, preserving SQLSTATE and reporting the schema and full history key without committing a partial backfill.

Batched pruning — migration 0028

• Add prune_executions_batch, which scans a primary-key-ordered page of 1–1,000 execution keys and returns a continuation cursor and scanned/deleted counts.
• Advance the cursor across protected executions, even when nothing is deleted.
• Preserve current/future executions, Running executions, and executions with NULL or insufficiently old completion timestamps.
• Precheck batch eligibility without locking, then lock only instances with deletion candidates. Both pruning APIs check eligibility under the same instance-row lock, preventing stale decisions during deletion/recreation or execution updates.
• Give atomic deletion a consistent instance-ID lock order to avoid multi-instance deadlocks with pruning.
• Remove the new function during schema cleanup, including cleanup of public.

Each batch is atomic. The limit bounds scanned execution keys, not the number of history events deleted.

Bulk deletion and compatibility

• Keep pending continuations out of age-filtered bulk instance deletion now that their completion timestamps are populated.
• Preserve existing unfiltered bulk-deletion behavior, retention options, and provider trait APIs.
• Include companion migration diffs, documentation, and regression coverage.

Upgrade notes

This requires a fully drained upgrade, not a rolling upgrade with old and new provider binaries together.

1. Pause all delete and prune jobs.
2. Stop and drain every process using the provider: workers, clients, retention and maintenance jobs, and direct SQL callers. Wait for in-flight calls and transactions to finish.
3. Apply the migrations with provider traffic stopped, then restart only new binaries containing these fixes and updated direct SQL callers.
4. Review retention cutoffs before resuming delete and prune jobs.

Stopping only workers is insufficient: old age-based deletion code can delete pending continuations after their completion timestamps are populated.

Expect a one-time wave of age-based deletions of old continue-as-new generations when retention resumes. Backfill time and WAL volume grow with the number of continue-as-new rows to repair; large histories can further increase backfill time. The backfill itself does not delete executions or history.

If the upgrade was not fully drained, an old acknowledgement can write a NULL completion timestamp after the backfill. Drain all callers first, then use the explicit timestamp repair procedure. Re-running the backfill while old calls remain in flight does not close that race; do not delete migration ledger rows to force a rerun.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The production data backfill and replacement of a critical acknowledgement function warrant the planned manual testing and human review.

Review effort: Balanced
Findings: None

What changed in this PR

Records completion timestamps for continued-as-new executions, enabling age-based pruning of older generations.

Changes:

• Updates acknowledgement logic and backfills recoverable timestamps.
• Adds regression coverage for migration and pruning behavior.
• Documents migration behavior and operational guidance.

File	Description
migrations/​0024_continue_as_new_completed_at.sql	Updates and backfills completion timestamps.
migrations/​0024_diff.md	Documents the migration delta.
migrations/​README.md	Adds upgrade and pruning guidance.
src/​migrations.rs	Refreshes the migration embedding marker.
tests/​regression_tests.rs	Tests fresh installs, upgrades, and pruning.
CHANGELOG.md	Records the fix under Unreleased.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[waldemort-auto]

Heads-up from #44, now tracked in #46.

#44 also adds migration 0026 and replaces ack_orchestration_item. It adds an advisory lock, a token recheck and a row-count check, to fix the double processing in #29. This PR's 0024 rebuilds ack_orchestration_item from the 0020 body. If this PR merges after #44 as it is, it silently reverts that fix.

Whichever PR merges second needs to:

1. renumber its migration;
2. carry both changes in its copy of ack_orchestration_item;
3. run cargo test --test deadlocks --test orchestration_ack_lease_tests.

[Copilot]

🟡 Changes recommended

A delete/recreate race can let pruning erase a replacement instance’s current execution and history, while regression tests do not independently verify key eligibility safeguards.

2 open issues

• Prevent delete/recreate race from pruning a replacement instance · New
• Strengthen pruning regression coverage for execution eligibility boundaries · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Batched pruning and atomic deletion acquire instance locks in inconsistent orders, risking deadlocks during concurrent multi-instance operations that existing single-instance tests cannot expose.

1 open issue

• Prevent deadlock from inconsistent instance lock ordering · New

2 resolved since last review

• Prevent delete/recreate race from pruning a replacement instance
• Strengthen pruning regression coverage for execution eligibility boundaries

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Concurrent batch and whole-instance pruning can deadlock without a shared instance-row lock, and schema cleanup leaves the new batch-pruning function behind.

2 open issues

• Prevent cross-API deadlocks with instance-row locking · New
• Drop prune_executions_batch during public-schema cleanup · New

1 resolved since last review

• Prevent deadlock from inconsistent instance lock ordering

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The retention backfill and destructive-operation lock-order changes require PostgreSQL-backed upgrade, replay/pruning, and concurrency validation that was unavailable in this review.

No open issues 🎉

2 resolved since last review

• Prevent cross-API deadlocks with instance-row locking
• Drop prune_executions_batch during public-schema cleanup

_{Review effort: Balanced}