[Thom Chiovoloni (thomcc-work)]

Summary

Add opt-in, nonblocking resource admission for activities, motivated by HTTP concurrency limits and policy enforcement in pg_durable. Activities that cannot acquire capacity can remain durably queued instead of occupying worker slots while waiting.

• Add register_with_admission() and register_typed_with_admission() for untyped and typed activity registration.
• Admit(guard) retains a resource reservation for the handler task's lifetime, releasing it on success, error, panic, or cancellation.
• Defer(delay) releases the worker slot while preserving the pending activity, without recording an activity result. A successfully committed deferral restores its fetch attempt.
• Clamp positive admission delays to 1 ms through 24 hours before they reach any provider. SQLite also saturates oversized relative timestamps rather than allowing them to wrap.
• Order eligible worker items by visibility time, then queue ID, so a deferred backlog cannot repeatedly overtake already-ready work. Add the supporting SQLite index through a migration for both new and existing file-backed databases.
• Record observed successful deferrals separately from activity executions, and emit the "Activity started" log only after admission succeeds.

Admission errors, panics, and zero delays intentionally produce a durable ActivityFailed application result with retryable: false, without invoking the handler. Defer is the signal for temporary backpressure; automatically retrying faulty admission logic could repeat invalid input or broken configuration indefinitely. The orchestration can catch the failure and choose recovery.

Existing registration APIs remain available, and activities without an admission callback admit by default. Regression tests cover typed codecs, deferral and restart recovery, guard cleanup, terminal rejection, delay bounds, backlog progress, metadata failures, public API compatibility, and file-backed database upgrades.

Checklist

• Tests added/updated
• Regular tests pass locally (cargo nt)
• Doc tests pass locally (cargo test --doc --all-features)
• Docs updated
  • Existing docs updated where behavior changed
  • New doc added under docs/ if introducing a new area — N/A; existing guides extended.
  • Linked from docs/README.md — N/A; this file is not present.

Links / Design notes

• PostgreSQL companion: Prevent deferred worker items from starving ready work duroxide-pg#51 provides the corresponding visibility-ordered worker fetch and index migration.
• Admission callbacks are synchronous and must not block or perform I/O. They must reserve all required capacity atomically or defer while holding none.
• The poison-message budget counts fetch attempts, not handler starts. Crashes, failed abandons, and metadata-read failures can consume attempts before a handler runs. A requested delay is not a durable "not before" guarantee until its abandon commits; lock expiry can make an uncommitted deferral eligible sooner.
• Activity metadata uses the lightweight instance lookup when it describes the requested execution. Providers without that capability, and activities from an earlier execution, read the full execution history on every attempt. If that fallback read fails, the worker abandons the fetch with its attempt counted and reports an infrastructure error rather than invoking the activity with unknown metadata. This applies to ordinary and admission-controlled activities.
• Resource coordination remains application-owned. Queue ordering allows already-ready work to progress past deferred items, but does not guarantee equal throughput or coordinate resource limits across runtimes.
• Event and work-item formats and provider method signatures remain unchanged. Completed results replay as before. Storage changes add supporting indexes; the existing public metrics snapshot shape is preserved.

Latest targeted validation

• Passed: 134 tests in each feature configuration

  ./run-tests.sh --test observability_tests --test activity_admission_tests --test shutdown_tests --test worker_reliability_test --test sqlite_tests --test registry_tests --test typed_composability_tests

• Passed: strict targeted Clippy

  cargo clippy --all-features --lib --test observability_tests --test activity_admission_tests --test shutdown_tests --test worker_reliability_test --test sqlite_tests --test registry_tests --test typed_composability_tests -- -D warnings

• cargo check --all-targets --all-features — passed
• cargo fmt --all -- --check — passed

[Copilot]

🟡 Changes recommended

Positive sub-millisecond deferrals are truncated to zero by SQLite, allowing an older item to be fetched repeatedly and starve ready activities without reaching the poison limit.

1 open issue

• Clamp positive sub-millisecond deferrals to one millisecond · New

What changed in this PR

Adds nonblocking activity admission control to Duroxide, allowing activities to reserve resources or defer execution without holding worker slots or recording activity results.

Changes:

• Introduces admission decisions and registry callbacks for resource reservation.
• Integrates durable deferral and handler-lifetime resource guards into worker execution.
• Documents admission semantics and adds regression tests for restart recovery, rejection, and guard cleanup.

File	Description
tests/​activity_admission_tests.rs	Tests deferral, restart recovery, rejection, and guard cleanup.
src/​runtime/​registry.rs	Adds activity registration with admission callbacks.
src/​runtime/​mod.rs	Defines admission decisions and the handler admission hook.
src/​runtime/​dispatchers/​worker.rs	Handles admission, durable deferral, and reservation lifetime.
docs/​ORCHESTRATION-GUIDE.md	Explains admission behavior, constraints, and semaphore usage.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Oversized defer delays can overflow SQLite timestamp conversion, making activities immediately eligible for repeated refetching without consuming their attempt budget.

1 open issue

• Prevent oversized defer delays from wrapping timestamps · New

1 resolved since last review

• Clamp positive sub-millisecond deferrals to one millisecond

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the activity admission-control changes.

No open issues 🎉

1 resolved since last review

• Prevent oversized defer delays from wrapping timestamps

_{Review effort: Balanced}

[Thom Chiovoloni (thomcc-work)]

Admission failures are intentionally terminal. Temporary capacity shortages should return Defer; Err represents permanent rejection, while panic and zero delay indicate faulty admission logic. These produce a durable ActivityFailed application result with retryable: false, rather than automatic worker retries that could repeat invalid input or broken configuration indefinitely. The orchestration can handle the failure and choose recovery.

[waldemort-auto]

Approving at 112dcf0 on behalf of Affan Dar (@affandar), after his review. Posted by the Waldemort agent.

Admission control, the deferral path, and the new SQLite worker fetch order look good, and the earlier review findings are fixed or documented.

Verified on the SQLite provider: the new validation test test_worker_deferred_item_does_not_starve_ready_work fails 3 of 3 on main with only the test added, and passes 3 of 3 at 112dcf0. Reverting either fetch-order change (src/providers/sqlite.rs:1849 session, :1870 plain) makes it fail again.

Follow-ups:

1. Docs: activities that can wait forever. See the inline comment on the admission section.
2. SQLite orchestration fetch order: filed as #77 (low priority, for consistency with PostgreSQL).

[waldemort-auto]

Please document what follows from this: an admission callback that never admits keeps the activity pending forever. Each committed deferral restores the attempt, so max_attempts is never reached, and the orchestration gets no result and no error. Correct-looking code can do this: a permit that leaks (a guard moved into a detached task, or a handler that hangs while holding it), a limit of 0, or a circuit breaker that never closes.

This is the second case in an existing category. The first is filtering: when no running worker can pick an item up, for example because no worker's TagFilter matches its tag (line 613), the item is never fetched and also waits forever.

Suggest documenting both in one place, with the same remedy:

• Wait on such activities with a timeout: select2 against a timer, or RetryPolicy::with_timeout. When the timeout wins, the losing activity is cancelled and its queued work item is deleted, which also ends a deferral loop.
• Alert when duroxide_activity_admission_deferrals_total keeps rising for one activity over a long period.