Add opt-in LISTEN/NOTIFY long polling

.github/workflows/test.yml

@@ -77,4 +77,6 @@ jobs:
         env:
           # Use the PostgreSQL service container for CI tests
           DATABASE_URL: postgres://postgres:postgres@localhost:5432/duroxide_test
-        run: cargo test --verbose
+        run: |
+          cargo test --verbose
+          DUROXIDE_PG_TEST_LONG_POLL=true cargo test --test postgres_provider_test --verbose

CHANGELOG.md

@@ -7,6 +7,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Opt-in long-polling through `ProviderConfig::long_poll`, for both URL and
+  Entra connections. It uses LISTEN/NOTIFY, bounded timer tracking for both
+  queues, periodic recovery of missed notifications, and reconnect handling.
+  Existing constructors remain short-polling by default.
+- Follow-up empty fetches and timer catch-up scans are coalesced at 100 ms
+  intervals; the first notification wake remains immediate. The configuration
+  type is non-exhaustive to permit compatible additions.
+- Acknowledgments notify other orchestration waiters after releasing a claim,
+  and per-queue catch-up scans wake work that became due between timer wakes.
+- Claims, renewals, and session leases announce future expiry deadlines, allowing
+  waiting dispatchers to recover crashed owners without waiting for the full
+  poll timeout. Delayed orchestration abandonment also wakes other visible work
+  immediately when it releases the instance lock.
+- Long-poll startup retains a warm fetch connection alongside LISTEN. Read-only
+  timer scans use 200 ms server-side lock and statement timeouts and return
+  their rolled-back connections to the pool. A one-second client watchdog
+  retains any pending scan instead of repeatedly detaching server backends;
+  degraded listeners keep polling independently of reconnect attempts.
+- Migration `0030` adds disabled notification triggers. An `ApplyAll` provider
+  with long-polling enabled explicitly activates them for its schema;
+  `VerifyOnly` requires prior activation and performs no DDL. Once activated,
+  queue and session-lease writes cannot use PostgreSQL `PREPARE TRANSACTION`.
+  Schemas that have not opted in retain their previous transaction behavior.
+- Long-polling reserves one connection within the configured pool and requires
+  at least two connections. Client-only providers can leave it disabled.
+  Notification channels support long and mixed-case schema names. This does
+  not convert the incompatible `duroxide-pg-opt` migration history.
+
+### Changed
+
+- Require SQLx 0.8.3 or later within the 0.8 series for explicit listener
+  reconnect handling. The existing native-TLS feature selection is unchanged.
+- Migration and notification-activation transactions use bounded lock waits
+  and jittered retries for deadlocks and lock contention, with a ten-second
+  retry budget. Permanent errors remain errors and failed attempts do not
+  change the migration ledger. Retry warnings include the SQLSTATE and full
+  error context.
+- Add a reproducible sparse/loaded workflow benchmark alongside the idle-query
+  benchmark, recording concurrency, duration, pool size, and commit settings.
+- Add a synchronous-standby contention profile with 16 orchestration and
+  16 worker slots, 64 clients, and concurrent lease-renewal measurements.
+  Benchmarks record sampled notification-commit and replication wait time;
+  documentation makes the per-claim and per-renewal notification costs explicit.
+
 ## [0.1.38] - 2026-10-26
 
 ### Fixed

Cargo.toml

@@ -32,7 +32,7 @@ exclude = [
 duroxide = { version = "0.1.32", features = ["provider-test"] }
 async-trait = "0.1"
 tokio = { version = "1", features = ["full"] }
-sqlx = { version = "0.8", features = ["runtime-tokio-native-tls", "postgres", "chrono"], default-features = false }
+sqlx = { version = "0.8.3", features = ["runtime-tokio-native-tls", "postgres", "chrono"], default-features = false }
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 dotenvy = "0.15"

README.md

@@ -174,9 +174,182 @@ Two test layers cover the Entra integration:
 |---------------------|-------------|---------|
 | `DUROXIDE_PG_POOL_MAX` | Maximum connection pool size | `10` |
 
+### Optional long-polling
+
+Existing constructors use short polling. To reduce idle queries and wake workers
+when work arrives, explicitly enable long-polling:
+
+```rust,no_run
+use duroxide_pg::{PostgresProvider, ProviderConfig};
+
+# async fn example(database_url: &str) -> anyhow::Result<()> {
+let mut config = ProviderConfig::url(database_url);
+config.schema_name = Some("my_app".to_string());
+config.long_poll.enabled = true;
+let provider = PostgresProvider::new_with_config(config).await?;
+# Ok(())
+# }
+```
+
+The same setting works with `ProviderConfig::entra(...)`. Long-polling reserves
+one connection **within** the provider's pool for PostgreSQL `LISTEN`, so the
+pool must allow at least two connections. It keeps another connection warm for
+fetches rather than leaving the first dispatcher to open one. Leave it disabled
+for client-only providers, especially one-connection backend clients. It requires a
+session-preserving database connection; transaction-pooling proxies are not a
+substitute for a dedicated LISTEN session.
+
+`LongPollConfig` also provides `notifier_poll_interval` (default 60 seconds)
+and `timer_grace_period` (default 100 milliseconds). The provider schedules
+future work and orchestration, activity, and session lease expiry, rescans after
+a timer fires, and periodically wakes fetches to recover missed notifications.
+Claims and renewals announce their lease deadlines without immediately waking
+fetches. An expired claim can therefore be recovered even if its owner crashes.
+Notifications are hints; queue predicates and locks remain authoritative.
+
+Read-only timer scans run in a transaction with 200 ms server-side lock and
+statement timeouts. PostgreSQL cancels blocked scans, and the rolled-back
+connection returns to the pool rather than leaving a blocked server backend
+behind. Scan errors retry after about one second.
+
+A one-second client watchdog keeps fetches checking for work if a scan stalls
+beyond those server limits. It does not cancel the scan or start another one;
+the same in-flight scan is retained across listener reconnects. Reconnecting
+also has an independent one-second polling fallback. Successful reconnection
+resubscribes and rechecks work.
+
+An enabled fetch waits for work until its `poll_timeout`; a zero timeout still
+performs a single fetch. The timeout bounds notification waiting, not an
+in-flight database operation. Dropping the provider or closing its pool stops
+the notification/timer task; connection cleanup has the network-failure
+limitation described below.
+The first notification wakes a fetch immediately. Further notifications that
+find no eligible work, and timer catch-up scans, are coalesced to a 100 ms
+cadence to bound query load. This can add up to 100 ms after an unsuccessful
+wake; the overall polling deadline is not extended.
+
+Migration `0030` installs **disabled** queue notification triggers. Constructing
+a provider with both `MigrationPolicy::ApplyAll` and `long_poll.enabled = true`
+activates them for that schema. `VerifyOnly` workers require that activation
+first and perform no DDL themselves. See [Migration Policy](docs/migration-policy.md).
+
+Once activated, inserts, visibility changes, lease changes, released locks, and
+consumption of claimed orchestration messages emit notifications. Acknowledgment
+notifications are delivered after the transaction releases the instance lock, so other
+workers recheck events that arrived during the turn. Delayed abandonment also
+announces the immediate instance-lock release, so other visible messages need
+not wait for the abandoned message's delay. This includes writes from
+short-polling producers. Turning off a provider's listener does not deactivate
+the schema's triggers. Transactions writing those queues or session leases cannot use
+`PREPARE TRANSACTION`, because PostgreSQL does
+not support preparing transactions that emit notifications. Schemas that have
+not opted in retain their existing transaction behavior.
+
+PostgreSQL serializes transactions that emit notifications through a
+server-wide lock held until commit completes. Slow WAL flushes or synchronous
+replication waits can therefore delay other notification writers, even in
+other schemas or databases on that server. This cost includes **every successful
+activity and orchestration claim**, renewals that change their lease deadlines,
+and session-lease writes. Empty fetches do not notify. A normal completed
+activity has three notifying commits (enqueue, claim, completion acknowledgment);
+an orchestration turn has two (claim and acknowledgment). Each lease-renewal
+transaction also joins this serialization when it emits a hint. Multiple hints
+in one transaction still acquire the commit lock only once.
+
+These are schema-wide costs, including short-polling callers of an activated
+schema. Benchmark concurrent writers with representative commit latency before
+activation; idle-query savings alone do not establish loaded throughput.
+
+Detected connection failures reconnect automatically, but silent network
+partitions are not actively probed. Until the transport reports a failure,
+work relies on the polling and refresh fallbacks instead of prompt
+notifications. SQLx listener cleanup can also wait indefinitely on a wedged
+socket, so provider drop does not guarantee bounded connection cleanup and
+`pool.close()` has no hard shutdown deadline in that situation.
+
+Recreate providers if their queue tables are dropped and recreated: notification
+channels identify the original tables. Runtime session-capacity decisions are
+also made once per fetch; a slot that started polling without session capacity
+may need another fetch after its poll timeout before it can claim session work.
+
+This feature does **not** convert a `duroxide-pg-opt` database into a
+`duroxide-pg` database. Their migration histories remain incompatible.
+
+#### Reproducing long-poll performance
+
+The ignored benchmarks in [tests/long_poll_tests.rs](tests/long_poll_tests.rs)
+compare short and long polling without changing the runtime's lease durations.
+Use a disposable, optimized PostgreSQL server with assertions disabled:
+
+```sh
+DUROXIDE_PG_POOL_MAX=10 cargo nextest run --release --test long_poll_tests \
+  --run-ignored only --nocapture \
+  -E 'test(workflow_latency_and_throughput_with_long_polling)'
+```
+
+Use otherwise idle, controlled hosts for comparisons. Concurrent workloads can
+invalidate both throughput comparisons and wait-time measurements.
+
+The workflow driver uses two orchestration slots, two worker slots, a 100 ms
+minimum poll interval, and a 30 s long-poll timeout. It measures 20 sparse
+one-activity workflows with a 200 ms idle gap, then 20 concurrent clients running
+one- and five-sequential-activity workflows for 10 seconds, with three loaded
+repetitions per mode. Activities return immediately. A completed warm-up workflow
+is excluded. Latency includes enqueue and the client's completion-status polling;
+throughput includes draining in-flight workflows after submissions stop.
+
+Each `BENCH` JSON line records the mode, pool limit, concurrency, duration,
+PostgreSQL version, commit settings (`fsync`, `full_page_writes`,
+`synchronous_commit`, and `synchronous_standby_names`), and replication state.
+Loaded runs sample notification-commit lock waits and `SyncRep` waits every
+10 ms. The reported backend-seconds are estimates integrated across the
+provider's connections, not individual-query latency measurements, and can
+exceed wall-clock time. The notification measurement uses PostgreSQL's shared
+`pg_database` object-0 heavyweight lock, not the `NotifyQueue` LWLock.
+Keep the settings and source revision alongside results; the driver does not
+relax durability.
+The separate `idle_query_load_is_reduced_with_long_polling` benchmark requires
+`pg_stat_statements` and measures 65 seconds per mode. Run it alone in an
+otherwise idle disposable database: it counts database-wide top-level statements
+except its statistics queries, including timer-scan transaction control.
+The `pg-stress` binary continues to use short polling.
+
+For commit-contention measurements, use a disposable primary with a physical
+synchronous standby. The higher-concurrency profile uses 16 orchestration
+slots, 16 worker slots, 64 clients, and a warmed pool; it also measures
+concurrent orchestration, activity, and session renewals with 60-second leases.
+Both workflow and renewal runs use three 10-second repetitions per mode.
+The renewal workload deliberately saturates renewal calls rather than using
+the runtime's normal renewal intervals.
+
+For a reproducible 10 ms apply-latency experiment, configure the standby's
+connection with `application_name=pr41_bench_standby` and set
+`recovery_min_apply_delay = '10ms'` there. On the primary, keep `fsync` and
+`full_page_writes` enabled and use:
+
+```conf
+synchronous_standby_names = 'FIRST 1 (pr41_bench_standby)'
+synchronous_commit = 'remote_apply'
+```
+
+Once that standby is streaming, run against the primary:
+
+```sh
+DUROXIDE_PG_POOL_MAX=100 cargo nextest run --release --test long_poll_tests \
+  --run-ignored only --nocapture \
+  -E 'test(notification_contention_with_synchronous_standby)'
+```
+
+The test rejects missing synchronous replication and insufficient pool capacity.
+It requires observed notification and replication waits in the enabled runs;
+disabled-schema controls must have no notification-lock waits. This experiment
+models a delayed synchronous apply acknowledgment, not a particular cloud
+service's replication topology.
+
 ## Features
 
 - Automatic schema migration on startup
+- Optional notification-driven long-polling for orchestration and activity work
 - Connection pooling via sqlx
 - Custom schema support for multi-tenant isolation
 - Full implementation of the Duroxide `Provider` and `ProviderAdmin` traits

docs/migration-policy.md

@@ -33,6 +33,71 @@ separate privileged process has already run migrations.
 `ApplyAll` takes a PostgreSQL session advisory lock scoped to the target schema
 before checking or applying migrations. That serializes concurrent provider
 startup so multiple nodes do not race to apply the same migration.
+Each migration transaction uses a 250 ms `lock_timeout`. Deadlocks (`40P01`) and
+lock timeouts (`55P03`) roll back and retry the entire transaction with jitter,
+within a ten-second contention budget per migration. Other errors return
+immediately. A failed attempt does not advance the migration ledger. The
+migration connection is closed on exit or cancellation so a failed unlock
+cannot leave a session advisory lock in the pool.
+
+The contention budget is currently fixed, not a provider configuration option.
+If it runs out, inspect the logged schema, SQLSTATE, and error chain, stop or
+drain competing workers and admin writers, and allow transactions holding the
+blocking locks to finish. Then retry `ApplyAll` from the process with DDL
+privileges. Earlier successful migrations remain recorded; the failed
+transaction is rolled back and will be retried on startup. Do not delete or
+edit migration ledger rows to force a retry. Backfill migrations must use their
+drained-upgrade procedure: a transaction retry repeats that backfill's work.
+
+## Long-Poll Notification Activation
+
+Migration `0030` installs disabled notification triggers. With
+`long_poll.enabled = true`, `ApplyAll` also enables the queue and session triggers in a
+transaction while holding the migration lock. `VerifyOnly` requires that the
+triggers are already active; otherwise construction returns an error explaining
+that an `ApplyAll` provider must activate them first. It never performs DDL.
+Activation follows acknowledgments' worker-before-orchestration lock order.
+Every lock acquisition, including the first queue, uses the same 250 ms
+`lock_timeout` and ten-second retry budget as migration transactions. Each
+failed attempt rolls back before the jittered delay. This bounds interference
+with writers and allows activation to outlast short bursts of reverse-order
+admin-delete contention. Exhausting the budget returns the database error
+without partially activating triggers.
+
+Activation is schema-wide and persists after the initializing provider exits.
+It allows short-polling clients to notify long-polling workers. Constructing a
+provider with long-polling disabled does not deactivate these shared triggers.
+Schemas that have not opted in do not emit notifications.
+
+PostgreSQL cannot prepare a transaction that emits a notification, so
+`PREPARE TRANSACTION` is not supported for queue or session-lease writes after activation.
+Reverting to two-phase-commit use requires stopping all long-polling workers
+and explicitly disabling all six notification triggers:
+
+- `worker_queue`: `trg_notify_worker_work`, `trg_notify_worker_work_released`
+- `orchestrator_queue`: `trg_notify_orch_work`, `trg_notify_orch_work_released`,
+  `trg_notify_orch_work_consumed`
+- `sessions`: `trg_notify_session_lease`
+
+These are verified and enabled from a single trigger inventory;
+changing a process's configuration alone is not sufficient.
+
+Notification-emitting commits also participate in PostgreSQL's server-wide
+notification-writer serialization. WAL or synchronous-replication delays can
+block other notification senders, including in different databases. Activation
+adds this cost to every successful queue claim, lease-changing orchestration
+and activity renewal, and session-lease write, not just enqueue and acknowledgment.
+A normal activity therefore has three notifying commits and an orchestration
+turn two; renewal transactions also notify. Empty fetches do not.
+Evaluate the loaded-write and renewal costs before opting a schema in;
+activation affects short-polling callers too. Disabling a process's listener
+does not remove the commit-lock cost of the schema's triggers. See the
+[synchronous-standby benchmark](../README.md#reproducing-long-poll-performance)
+for a higher-concurrency measurement that records notification-lock wait time.
+
+This is an additive migration within the `duroxide-pg` schema lineage, not a
+conversion of `duroxide-pg-opt` state. Older binaries still reject schemas with
+unknown migration versions, even when notifications remain disabled.
 
 ## Error Messages