Skip to content

Fix outbox stuck with multi-tenant RabbitMQ and durable messaging#2364

Merged
jeremydmiller merged 1 commit intomainfrom
fix-outbox-multi-tenant-stuck
Mar 28, 2026
Merged

Fix outbox stuck with multi-tenant RabbitMQ and durable messaging#2364
jeremydmiller merged 1 commit intomainfrom
fix-outbox-multi-tenant-stuck

Conversation

@jeremydmiller
Copy link
Copy Markdown
Member

Summary

  • Root cause: TenantedSender implemented ISenderRequiresCallback, which caused SendingAgent to use sendWithCallbackHandlingAsync. That code path assumes the inner sender calls back on success via ISenderCallback, but RabbitMqSender (and all other transport senders used with TenantedSender) does NOT implement ISenderRequiresCallback. So DurableSendingAgent.MarkSuccessfulAsync() was never called after a successful send, the outbox entry was never deleted, and the durability agent re-sent messages every 5 seconds causing duplicate envelope exceptions.
  • Fix: Remove ISenderRequiresCallback from TenantedSender. This makes SendingAgent use sendWithExplicitHandlingAsync which explicitly calls MarkSuccessfulAsync(envelope) after successful sends, properly deleting the outbox entry. Also fixes Dispose() to dispose all IDisposable inner senders (not just ISenderRequiresCallback ones).
  • Impact: This bug affects all transports using TenantedSender with durable outbox — RabbitMQ, NATS, and Azure Service Bus multi-tenancy configurations.

Fixes #2361

Test plan

  • New reproducing test Bug_2361_outbox_stuck_with_tenanted_broker passes (was failing before fix — 2 stuck outbox messages, now 0)
  • All 5 TenantedSenderTests pass (unit tests for TenantedSender behavior)
  • Bug_475_durable_outbox_sending_out_of_order passes (durable outbox regression)
  • Bug_2304_conventional_routing_ignores_durable_outbox_policy passes
  • All multi-tenancy internal tests pass
  • Full RabbitMQ test suite: 269 passed, 8 failed (all 8 are pre-existing SQL Server connection failures in Bug_1594_ReplayDeadLetterQueue and Bug_DLQ_NotSavedToDatabase)

🤖 Generated with Claude Code

…le outbox

TenantedSender implemented ISenderRequiresCallback, which caused
SendingAgent to pick the sendWithCallbackHandlingAsync code path. That
path assumes the inner sender calls back on success via the registered
ISenderCallback. But RabbitMqSender (and all other transport senders
used with TenantedSender) does NOT implement ISenderRequiresCallback —
it is a simple fire-and-forget sender. So after a successful send
through a tenanted endpoint, DurableSendingAgent.MarkSuccessfulAsync()
was never called, the outbox entry was never deleted, and the durability
agent kept re-sending the message every 5 seconds.

The fix removes ISenderRequiresCallback from TenantedSender. This makes
SendingAgent use sendWithExplicitHandlingAsync instead, which explicitly
calls MarkSuccessfulAsync after a successful send, properly cleaning up
the outbox entry. This also fixes the Dispose() method to dispose ALL
IDisposable inner senders, not just ISenderRequiresCallback ones.

Fixes #2361

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Message is stuck in outbox with multiple tenants and durable messaging

1 participant