[Thom Chiovoloni (thomcc-work)]

Fixes #139.

Summary

Adds administrator-controlled admission quotas to df.start() to limit runaway workflow submission and retained history.

Setting	Default	Limits
df.max_concurrent_per_user	100	Queued and nonterminal workflow instances
df.max_instances_per_user	10,000	Retained submissions, including completed, failed, cancelled, and rolled-back starts

Both settings are superuser-settable, apply per submitting role and origin database, and accept 0 to disable the individual limit. Superuser submission identities are exempt, subject to the existing superuser-instance policy. Rejected starts raise SQLSTATE 53400 with an actionable hint.

These are concurrency and retention quotas, not calls-per-second or byte limits.

Implementation

• Race-safe admission: Serialize starts for the same role/database until the submitting transaction ends. Use current snapshots so older repeatable-read or serializable snapshots cannot bypass the limits. Internal lock tags prevent unrelated roles from acquiring another role’s admission lock through SQL advisory-lock functions.
• Protected accounting: Determine active capacity from engine state rather than caller-writable display status. Cancellation frees active capacity after the engine processes it.
• Rollback-resistant submissions: Commit a protected submission record atomically with engine enqueue. Transaction and savepoint rollbacks discard the graph but cannot erase the quota charge for engine work already submitted.
• Safe cleanup: Release retained accounting only after engine work, history, and origin metadata are gone. Failed metadata deletion preserves retirement evidence without making retired work count as active again.
• Collision handling: Retry IDs reserved in either the local instance table or the protected ledger, including IDs retained after rollback. Failed persistence or enqueue releases uncommitted reservations.
• Apply the same enforcement to both transaction modes. 'new' uses its separate session’s configuration defaults rather than inheriting caller session overrides.

Upgrade & compatibility

Prepares version 0.2.10 and adds an owner-leading index through the extension upgrade script. Quotas also work with supported older schemas without ALTER EXTENSION UPDATE; the worker provisions protected accounting before publishing readiness. Updating the extension adds the index.

No orchestration payloads or replay-visible operations change. However, the new defaults also affect existing workflows whose SQL calls df.start().

Retained capacity is not freed by completion, cancellation, or rollback alone. With default retention, a role holding 10,000 recent submissions may remain blocked until history ages out or an administrator adjusts policy. Existing orphan records created before protected accounting cannot be attributed retroactively.

Validation

Local PG17 validation passed:

• 458 unit tests, with 16 existing skips.
• 15 targeted E2E tests covering quotas, concurrent transactions, rollback/savepoints, status forgery, lock isolation, cleanup failures, and satellite databases.
• All 177 upgrade checks, including supported schemas and the previous-binary lifecycle.
• Formatting, Clippy, pgspot, and all 9 Docker-runner checks.

Also fixes Docker’s test classification so reconciliation-only regressions run under the primary CI runner’s required configuration rather than the fixed-configuration Docker image.

[Copilot]

🟡 Changes recommended

A non-superuser can bypass both quotas through repeated transaction rollbacks because instance accounting is rolled back while enqueue commits separately, allowing unaccounted engine work and retained failure history.

1 open issue

• A non-superuser can repeatedly run BEGIN; SELECT df.start('SELECT 1'); ROLLBACK; to bypass both… · New

What changed in this PR

Adds per-role instance quotas to df.start() to address denial-of-service risks from excessive workflow submissions, with enforcement in both transaction modes.

Changes:

• Adds configurable active and retained-instance quotas, serialized admission checks, and quota error hints.
• Adds an owner-leading index and prepares version 0.2.10 with upgrade support.
• Adds quota, concurrency, isolation, and security tests, plus configuration and upgrade documentation.

File	Description
USER_GUIDE.md	Documents quota configuration, scope, retention, and transaction interactions.
tests/​e2e/​sql/​93_start_limits_serializable.sql	Tests overlapping serializable starts and transaction retries.
tests/​e2e/​sql/​92_start_limits_trust.sql	Tests protected accounting and resistance to status and lock manipulation.
tests/​e2e/​sql/​91_start_limits_concurrency.sql	Tests admission locking, isolation levels, rollback, and timeouts.
tests/​e2e/​sql/​90_start_limits.sql	Tests quota settings, enforcement, identities, and transaction modes.
tests/​e2e/​sql/​89_schema_qualification.sql	Adds aggregate-shadowing traps for quota queries.
tests/​e2e/​sql/​00_setup_playground.sql	Adds a helper to wait for engine-terminal state.
src/​lib.rs	Registers quota settings and the fresh-install index.
src/​dsl.rs	Enforces quotas with per-role locking and current-snapshot counts.
src/​client.rs	Counts engine-active instances and preserves quota error details.
sql/​pg_durable--0.2.9--0.2.10.sql	Adds the quota index during extension upgrade.
scripts/​test-upgrade.sh	Tests quota enforcement against older installed schemas.
docs/​upgrade-testing.md	Describes quota compatibility, migration, and performance considerations.
docs/​api-reference.md	Documents quota settings and start rejection behavior.
CHANGELOG.md	Records quota security changes and upgrade implications.
Cargo.toml	Bumps the package version to 0.2.10.
Cargo.lock	Aligns the locked package version with 0.2.10.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Submission records can be deleted despite failed local metadata cleanup, falsely blocking submissions through quota accounting, while reused rolled-back IDs can abort starts instead of triggering retries.

2 open issues

• Preserve submission accounting when metadata deletion fails · New
• Retry IDs reserved by rolled-back submissions · New

1 resolved since last review

• A non-superuser can repeatedly run BEGIN; SELECT df.start('SELECT 1'); ROLLBACK; to bypass both…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Security-sensitive admission combines PostgreSQL locking and snapshots with independently committed provider state, requiring human review and unavailable runtime validation of concurrency, rollback, cleanup, and binary-only upgrades.

No open issues 🎉

2 resolved since last review

• Preserve submission accounting when metadata deletion fails
• Retry IDs reserved by rolled-back submissions

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Human review is needed to verify unavailable upstream provider atomicity contracts and validate PostgreSQL concurrency, lock ordering, and cross-database failure recovery.

No open issues 🎉

_{Review effort: Balanced}