Skip to content

fix(asb): de-quadratic session listeners, surface MaxConcurrentCalls, settle the original on a batched defer (GH-3494) - #3674

Merged
jeremydmiller merged 1 commit into
mainfrom
perf/3494-asb-session-and-settlement
Jul 28, 2026
Merged

fix(asb): de-quadratic session listeners, surface MaxConcurrentCalls, settle the original on a batched defer (GH-3494)#3674
jeremydmiller merged 1 commit into
mainfrom
perf/3494-asb-session-and-settlement

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

The Azure Service Bus deep dive (GH-3494), Wave 0 — the items the plan calls out as landable from
code review and tests, before a real namespace is available. No emulator timings are quoted:
§2 and §8 rule the emulator out as a source of publishable numbers, and AE2's PrefetchCount
validation still needs a real Standard/Premium namespace. Everything below is an exact count.

AO2 — the n² was real, on both session paths

ListeningAgent builds Endpoint.ListenerCount listeners whenever that count exceeds 1
(ListeningAgent.cs:373-381). Both session implementations then multiplied by ListenerCount
again.

Accept-loop listener (AzureServiceBusSessionListener, the default when
ConfigureSessionProcessor is unset) — concurrent AcceptNextSessionAsync loops:

RequireSessions(n) Before After
unset / 1 1 1
3 9 3
5 25 5
10 100 10

ServiceBusSessionProcessor path (opt-in, added in #3533) — I did not expect to find this one.
BuildSessionProcessorOptions set MaxConcurrentSessions = endpoint.ListenerCount on each of the
ListenerCount processors:

RequireSessions(n) Concurrent sessions before After
8 8 processors x 8 sessions = 64 8

Both now yield exactly n, which is what RequireSessions(n) is documented to mean, and a user
ConfigureSessionProcessor(o => o.MaxConcurrentSessions = ...) still overrides.

⚠️ Reviewer's call: this changes behavior that build_session_processor_options_maps_listener_count_to_max_concurrent_sessions
asserted in #3533, merged four days ago. I rewrote that test to assert the total-concurrency
invariant instead, with the reasoning inline, and added one covering the user override. If #3533
intended 8x8 deliberately, say so and I'll revert this half.

AO3 — inline concurrency was 1, and unreachable

Wolverine never set ServiceBusProcessorOptions.MaxConcurrentCalls, so every inline ASB endpoint ran
at the SDK default of one concurrent handler invocation, changeable only through the raw
ConfigureProcessor hook (MaximumParallelMessages does not apply to inline endpoints — it sizes
Wolverine's in-process worker queue, which inline bypasses).

opts.ListenToAzureServiceBusQueue("orders").ProcessInline().MaximumConcurrentCalls(10);

Applied before ConfigureProcessor so that hook still wins. On the session-processor path it maps to
MaxConcurrentCallsPerSession and stays at 1 unless explicitly asked for, since raising it discards
the per-session FIFO ordering that is usually the point of sessions.

AO8 — one deferral produced two deliveries

The batched listener's _defer block re-sent a copy but never settled the original, which then sat
locked until the lock expired and Azure Service Bus redelivered it.

Before After
Deliveries from one deferral 2 (re-sent copy + redelivered original) 1

InlineAzureServiceBusListener already did complete-then-resend; the batched path now matches.

AO4 — deferred, needs an upstream change

Widening settlement concurrency is not a Wolverine-side change: RetryBlock hard-codes
new Block<Item>(1, Block<Item>.Unbounded, executeAsync), so the per-message CompleteMessageAsync
calls are serialized by construction. Noted in the plan.

Tests

12 new unit tests (accept-loop count across RequireSessions 0/1/5/10; the MaximumConcurrentCalls
surface, its precedence against ConfigureProcessor, and the session-processor mapping), plus the
two rewritten session-processor tests.

The emulator suite is noisy on this box, so every failure was baselined against a clean
origin/main worktree on the same emulator with the same filter:

Run Build Result
Full suite this branch 271 passed / 25 failed (1h01m)
Failing subset (34 tests) this branch 7 passed / 27 failed (1h04m)
Same subset clean origin/main 11 passed / 23 failed (59m)
2 session tests, isolated, x3 this branch 0 of 6 passed
2 session tests, isolated, x3 clean origin/main 0 of 6 passed
when_using_handler_type_naming, isolated both 2/2 passed on each

Two failure modes, both environmental and both reproduced on clean main:

  1. ConventionalRouting / conventional-routing BugsBrokerInitializationException: Unable to initialize the Broker asb in time. Emulator broker-init timeouts under the per-class
    DeleteAllEmulatorObjects + AutoProvision churn. The count moves around with test ordering
    (21 in the full run, 25 in the isolated subset, 23 on main) — it goes up when fewer tests run,
    which is the opposite of a code regression. Nothing in this diff touches discovery or provisioning.
  2. Two session ordering assertions (["One","Three","Two"], ["Dummy 4.2","Dummy 4.1"]) — these
    passed in main's 34-test subset and failed in the branch's, which looked branch-specific until I
    ran them in isolation: 3/3 failures on branch and 3/3 on clean main. The earlier main pass was
    test-ordering luck. This matches the plan's own §8 note that sessions misbehave on the emulator.
    Both endpoints use RequireSessions() / RequireSessions(1), so AO2 spawns one accept loop
    before and after — that path is unchanged for them.

Operational note for anyone rerunning these: AzureServiceBusTesting calls
DeleteAllEmulatorObjectsAsync() once per test process, so two ASB test processes against one
emulator wipe each other's entities. These runs have to be serial.

Still open on GH-3494

AO1 (ServiceBusSessionProcessor rewrite of the accept-loop listener), AO4, AO5 (lock renewal on the
batched path), AO6/AE2 (PrefetchCount validation and guidance), and the whole real-namespace matrix.

🤖 Generated with Claude Code

… settle the original on a batched defer (GH-3494)

AO2 -- ListeningAgent already builds Endpoint.ListenerCount listeners, and
BOTH session paths then multiplied by ListenerCount again.
AzureServiceBusSessionListener spawned ListenerCount accept loops per
instance, so RequireSessions(5) opened 25 concurrent AcceptNextSessionAsync
loops competing for the same sessions. The ServiceBusSessionProcessor path
added in GH-3533 set MaxConcurrentSessions = ListenerCount on each of the
ListenerCount processors, so RequireSessions(8) ran 64 concurrent sessions.
Both now yield exactly n, which is what RequireSessions(n) documents. A user
ConfigureSessionProcessor customization still overrides.

AO3 -- Wolverine never set ServiceBusProcessorOptions.MaxConcurrentCalls, so
every inline Azure Service Bus endpoint ran at the SDK default of one
concurrent handler invocation, reachable only through the raw ConfigureProcessor
hook (MaximumParallelMessages does not apply to inline endpoints). New
MaximumConcurrentCalls(n) on the queue and subscription listener
configurations, applied before ConfigureProcessor so that hook still wins. On
the session processor path it maps to MaxConcurrentCallsPerSession and stays
at 1 unless asked for, since raising it discards per-session FIFO ordering.

AO8 -- the batched listener's defer block re-sent a copy without settling the
original, which then sat locked until the lock expired and was redelivered:
one deferral, two deliveries. Now completes first, matching the inline listener.

AO4 (settlement concurrency) is deferred -- RetryBlock hard-codes a parallel
count of 1, so that is a JasperFx change, not a Wolverine one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant