Repository navigation
GH-5171 Coalesce durable outbox/inbox writes per tenant database - #5180
Merged
Merged
Conversation
The durable sending agent's store and delete coalescers were shared by every tenant sending to one destination and ran one flush at a time, so one tenant's slow database delayed every other tenant's MarkSuccessfulAsync and outbox store (and the requests awaiting them). When the outbox or inbox can reach more than one database (MultiTenantedMessageStore, or the ancillary-store delegating wrappers), the coalescer now keeps one lane per database, keyed by the resolved tenant store via a lock-free ImHashMap lookup. Single-database stores keep the plain EnvelopeStoreCoalescer with no extra lookup. The durable local queue's inbox store coalescer gets the same treatment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
Merged
This was referenced Oct 25, 2026
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The durable sending agent's
_storeCoalescerand_deleteCoalescer(one per destination) were shared by every tenant and ran one flush at a time. With a database per tenant, one tenant's slow database (an exhausted pool, for example) delayed every other tenant'sMarkSuccessfulAsyncand outbox store, and with them the requests awaiting them.PartitionedEnvelopeStoreCoalescerkeeps one plainEnvelopeStoreCoalescerlane per database, in anImHashMap<object, EnvelopeStoreCoalescer>. Lookups are lock-free; a lane is added under a lock the first time a database is seen. Each lane flushes on its own, so a slow tenant database only delays its own envelopes.EnvelopeStoreCoalescing.Build/PartitionerForpick it only when the outbox/inbox can reach more than one database:MultiTenantedMessageStore,DelegatingMessageOutboxandDelegatingMessageInbox(ancillary stores). Any other store keeps the plain single-lane coalescer, so single-tenant behaviour doesn't change and there is no extra lookup.MultiTenantedMessageStore.CoalescingPartitionFor, a sync_byTenantlookup; the default tenant,*DEFAULT*andmainall go toMain). Tenants that share a database still share a lane and a batch. If a tenant's store hasn't been resolved yet, its lane is keyed by the tenant id until the first write caches the store. The delegating wrappers route byEnvelope.Storefirst, as their writes do.Judgment call: a lane keyed by tenant id before the store is cached stays in the map (empty, no loop running) once later envelopes move to the store-keyed lane. That is one small object per tenant per endpoint.
Tests run locally (net9.0, Release)
per_tenant_outbox_coalescing_5171(CoreTests, 5 tests): tenant A's deletes and stores are held open, and tenant B'sMarkSuccessfulAsync/TryStoreOutgoingAsyncstill complete right away. Also checks that tenants sharing a database still share one batch, that single-database stores keep the plain coalescer, and how the partition keys resolve. With the fix turned off, the two slow-tenant tests fail (5 s timeout).CoreTests.Persistence+Transports.Local+DurableSendingAgent: 395/395 passed. This coverscoalesced_*_4319,mixed_tenant_inbox_batches_4435,single_store_inbox_batches_4658andtenant_write_brake_4659.PostgresqlTests.MultiTenancy: 25/25 passed.Closes #5171
🤖 Generated with Claude Code
https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U