Skip to content

GH-5171 Coalesce durable outbox/inbox writes per tenant database - #5180

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5171-per-tenant-coalescer
Oct 23, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5171-per-tenant-coalescer

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

What changed

The durable sending agent's _storeCoalescer and _deleteCoalescer (one per destination) were shared by every tenant and ran one flush at a time. With a database per tenant, one tenant's slow database (an exhausted pool, for example) delayed every other tenant's MarkSuccessfulAsync and outbox store, and with them the requests awaiting them.

  • New PartitionedEnvelopeStoreCoalescer keeps one plain EnvelopeStoreCoalescer lane per database, in an ImHashMap<object, EnvelopeStoreCoalescer>. Lookups are lock-free; a lane is added under a lock the first time a database is seen. Each lane flushes on its own, so a slow tenant database only delays its own envelopes.
  • EnvelopeStoreCoalescing.Build/PartitionerFor pick it only when the outbox/inbox can reach more than one database: MultiTenantedMessageStore, DelegatingMessageOutbox and DelegatingMessageInbox (ancillary stores). Any other store keeps the plain single-lane coalescer, so single-tenant behaviour doesn't change and there is no extra lookup.
  • The partition key is the resolved tenant store (MultiTenantedMessageStore.CoalescingPartitionFor, a sync _byTenant lookup; the default tenant, *DEFAULT* and main all go to Main). Tenants that share a database still share a lane and a batch. If a tenant's store hasn't been resolved yet, its lane is keyed by the tenant id until the first write caches the store. The delegating wrappers route by Envelope.Store first, as their writes do.
  • The durable local queue's inbox store coalescer has the same shape, so it gets the same change.

Judgment call: a lane keyed by tenant id before the store is cached stays in the map (empty, no loop running) once later envelopes move to the store-keyed lane. That is one small object per tenant per endpoint.

Tests run locally (net9.0, Release)

  • New per_tenant_outbox_coalescing_5171 (CoreTests, 5 tests): tenant A's deletes and stores are held open, and tenant B's MarkSuccessfulAsync / TryStoreOutgoingAsync still complete right away. Also checks that tenants sharing a database still share one batch, that single-database stores keep the plain coalescer, and how the partition keys resolve. With the fix turned off, the two slow-tenant tests fail (5 s timeout).
  • CoreTests CoreTests.Persistence + Transports.Local + DurableSendingAgent: 395/395 passed. This covers coalesced_*_4319, mixed_tenant_inbox_batches_4435, single_store_inbox_batches_4658 and tenant_write_brake_4659.
  • PostgresqlTests PostgresqlTests.MultiTenancy: 25/25 passed.

Closes #5171

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

The durable sending agent's store and delete coalescers were shared by every
tenant sending to one destination and ran one flush at a time, so one tenant's
slow database delayed every other tenant's MarkSuccessfulAsync and outbox store
(and the requests awaiting them). When the outbox or inbox can reach more than
one database (MultiTenantedMessageStore, or the ancillary-store delegating
wrappers), the coalescer now keeps one lane per database, keyed by the resolved
tenant store via a lock-free ImHashMap lookup. Single-database stores keep the
plain EnvelopeStoreCoalescer with no extra lookup. The durable local queue's
inbox store coalescer gets the same treatment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit 8ca03f1 into main Oct 23, 2026
@jeremydmiller jeremydmiller mentioned this pull request Oct 23, 2026
This was referenced Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Durable outbox: one tenant's slow database holds back every tenant's sends (the delete coalescer is shared)

1 participant