[jeremydmiller]

Closes #5105

Mechanism

On a durable listener, MessageContext.MoveToDeadLetterQueueAsync hands the envelope to DurableReceiver.MoveToErrorsAsync. That method only posted the write to a background RetryBlock and returned. MoveToErrorQueue then completed the envelope, which marked the inbox row handled, and recorded MovedToErrorQueue. The broker delivery had already been settled when the inbox row was written, so that row was the only copy. If the write kept failing (a database outage, or a tenant database that could not be resolved as in the report), the block discarded it after 3 attempts with a line at Information. No dead-letter row existed, yet the logs, tracker, wolverine.error.queued and the dead-letter counter all said the message had been dead lettered.

Fix (DurableReceiver, MessageContext)

• The write is awaited. MoveToErrorsAsync makes the dead-letter write (or the _deadLetterSender send) inline, retrying with pauses of 50ms, 250ms, 1s and 2s.
• A failed write is not reported as dead lettered. It throws DeadLetterMoveFailedException, the same contract NATS uses since 6.50.0, so MoveToErrorQueue and NoHandlerContinuation neither complete the envelope, publish the fault, nor record MovedToErrorQueue. They record Requeued instead. The failure is logged at Error with the envelope id, message type, tenant id and listener URI.
• The row is handed back. The envelope is rescheduled for any node after 30s (ScheduleExecutionAsync, or ScheduleRetryAsync on a database-backed endpoint), so it is executed again and dead lettered once the store returns. Otherwise the row would sit Incoming and owned by a live node, which nothing reclaims (An envelope whose completion could not be written stays owned by the live node until the node stops (6.41.0) #4664). The reschedule retries until the store comes back or the host stops, the same way mark-as-handled has since An envelope whose completion could not be written stays owned by the live node until the node stops (6.41.0) #4664.
• Envelopes dead lettered on receipt (unwrap failure or missing message type) are settled with the broker only after the write. If the write fails, they are deferred back to the broker.
• Batches: every member is now attempted before the failure surfaces. Stopping at the first failure left later members unmoved and stranded.
• Side fix: the missing-message-type receipt path built new ErrorReport(envelope, envelope.Failure!) with a null Failure, which threw a NullReferenceException inside the block, so those envelopes were never dead lettered either.

Reproduction

• CoreTests/Runtime/WorkerQueues/dead_letter_write_failures_5105.cs has 8 unit tests with a substituted inbox. On unmodified main, 6 of the 8 fail. The key one shows MarkIncomingEnvelopeAsHandledAsync called on an envelope whose dead-letter write had failed, and nothing surfacing. The two that pass on main are a control and a regression guard.
• PostgresqlTests/Transport/Bug_5105_failed_dead_letter_write.cs is end to end: a PostgreSQL durable queue listener with a handler that always throws and MoveToErrorQueue. The test renames wolverine_dead_letters away while the message fails, then renames it back. On main the dead-letter row never appears (the message was lost). With the fix the message runs again and lands in dead-letter storage, in about 40s.

Tests run locally (-f net9.0)

• CoreTests full suite: 3739 passed, 0 failed, 8 skipped
• PostgresqlTests full suite: 868 passed, 2 failed, 2 skipped. Bug_1942...inline_rabbitmq_listener_replay_does_not_loop failed because RabbitMQ was not running locally. inbox_outbox_usage.cascaded_response_with_outbox timed out under load and passed 3/3 on rerun; it does not touch the dead-letter path.
• dotnet build wolverine.slnx -c Release -f net9.0 builds clean

Follow-ups (not in this PR)

• Multi-tenant fallback: when a tenant store cannot be resolved, MultiTenantedMessageStore has no fallback for the dead-letter write. A fallback to Main would need the row's TenantId cleared, because MarkIncomingEnvelopeAsHandledAsync also routes by TenantId, as the reporter found. With this PR such a message is retried rather than lost.
• IDeadLetterInterceptor gap: the native-DLQ branch of MessageContext.MoveToDeadLetterQueueAsync, which a DurableReceiver takes, does not run IDeadLetterInterceptor. Only the non-native Storage.Inbox branch does. Interceptors therefore don't apply to durable listeners' dead-letter rows today. This is pre-existing and worth its own issue.
• The 30s redelivery delay and the inline retry budget are internal constants. Exposing them on DurabilitySettings would be an API decision.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U