Skip to content

GH-5107 PostgreSQL queue sender moves the outbox row atomically for every durable send - #5109

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5107-pg-queue-atomic-outbox-move
Oct 21, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5107-pg-queue-atomic-outbox-move

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5107

Mechanism

PostgresqlQueueSender.SendAsync(Envelope) only took the atomic MoveFromOutgoingToQueueAsync path (queue insert + outbox delete in one statement batch) when Envelope.WasPersistedInOutbox was true. Only the RDBMS MessageDatabase store paths set that flag. Envelopes whose outbox row came from the Marten outbox (StoreOutgoingEnvelope queued on the session), the EF Core outbox, or outbox recovery (ReadOutgoingAsync) arrive without it. Those went through the direct insert, and DurableSendingAgent deleted the outbox row afterwards, in a separate transaction.

If a node died between those two commits, it left the outbox row behind. Orphan release and outbox recovery then sent it again. The durable pop's GH-4316 inbox check drops that copy only while the receiver's Handled row still exists (KeepAfterMessageHandling, 5 minutes by default). After that, the copy was handled a second time.

Fix

For a durable endpoint, when the envelope is not flagged, the sender now uses a single statement:

WITH moved AS (DELETE FROM <storage>.wolverine_outgoing_envelopes WHERE id = :id RETURNING id, body, message_type, deliver_by)
INSERT INTO <queue> (id, body, message_type, keep_until)
SELECT ... FROM moved
UNION ALL
SELECT :id, :body, :type, :expires WHERE NOT EXISTS (SELECT 1 FROM moved)
ON CONFLICT (id) DO NOTHING;

The scheduled table has the same shape, with ON CONFLICT DO UPDATE to match the existing direct scheduled upsert. A crash now leaves either the outbox row with nothing on the queue, or the queued row with no outbox row. When there is no outbox row, the statement is just the old insert. The later DurableSendingAgent delete becomes a no-op.

Tests

New file: PostgresqlTests/Transport/Bug_5107_outbox_row_survives_the_queue_insert.cs

  • an_envelope_is_handled_once_when_the_sender_dies_after_the_queue_insert: an end-to-end version of the issue's scenario. It stores the outbox row under a "dead" owner and sends it unflagged, as Marten or recovery would. It waits for handling, deletes the handled inbox row to simulate expiry, then sets owner_id = 0 to simulate the orphan sweep and lets recovery run. Before the fix it fails expected 1 but was 2, with the same Recovered 1 messages from outbox log lines as the issue.
  • the_queue_insert_takes_the_outbox_row_with_it: before the fix, the outbox row was still there after the send (1 instead of 0).
  • a_scheduled_send_takes_the_outbox_row_with_it and an_envelope_with_no_outbox_row_is_still_written_to_the_queue: cover the scheduled table and the fallback insert.

Local results (net9.0):

  • PostgresqlTests.Transport namespace: 171 passed, 2 skipped, 0 failed.
  • Full PostgresqlTests: 870 passed, 2 skipped, 3 failed.
    • Two failures need RabbitMQ on 5672, which was not running locally.
    • inbox_outbox_usage.cascaded_response_with_outbox timed out once at 5s under full-suite load. It uses non-durable queues, so the changed path is not involved, and it passed 5 of 5 times on its own.
  • MartenTests PostgreSQL-queue classes (Bug_1175_schema_name_with_queues, multi_tenancy_queue_usage, using_tenant_specific_queues_and_subscriptions): 5 of 5 passed.
  • dotnet build wolverine.slnx -c Release -f net9.0: clean.

Not in this PR

  • The SQL Server, MySQL, Oracle and SQLite queue senders use the same WasPersistedInOutbox gate, so they very likely have the same window when used with the EF Core outbox or outbox recovery. They are not changed here.
  • A flagged send whose first attempt committed but lost its reply still hits No matching outgoing envelope on retry. That is existing behaviour and is unchanged here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

…very durable send

An envelope whose outbox row was written by the Marten or EF Core outbox, or read back by outbox
recovery, does not carry WasPersistedInOutbox, so the durable PostgreSQL queue sender inserted it into
the queue in one transaction and left DurableSendingAgent to delete the outbox row in a second. A node
dying in between left a row that recovery sent again; once the receiver's handled inbox row had expired
the copy was handled a second time.

Durable sends without the flag now go through a single statement that deletes the outbox row (if any)
and inserts it into the queue or scheduled table, falling back to the envelope's own body when there is
no outbox row. Sends with the flag keep the existing MoveFromOutgoingTo* path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit 388d266 into main Oct 21, 2026
47 checks passed
This was referenced Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostgreSQL queue: an envelope sent from a Marten outbox can be handled twice when the outbox sends it again after the receiver's handled row expired

1 participant