GH-4049: ASB native scheduling, and NativeAck refused with sessions

[jeremydmiller]

References #4049. Not "Closes" — see Scope below.

Read this first: #4049 was re-scoped, and two of its original premises were wrong

The issue was filed as "NativeAck needs a new listener class for ASB and Pub/Sub". Investigation against origin/main disproved two of its premises, and the investigation comment on #4049 re-scoped it. This PR follows that comment, not the original body. The corrections, so this PR does not look like it is ignoring its own issue:

1. The AutoCompleteMessages claim was wrong. The issue said Azure Service Bus: inline processor leaves AutoCompleteMessages at the SDK default (true), so a failed dead-letter/defer can be auto-completed behind Wolverine's back #4018 turns it off on the inline processor and the batched path does not set it. BuildReceiverOptions builds a ServiceBusReceiverOptions, which has no AutoCompleteMessages property at all — that belongs to the processor options. The batched path is peek-lock and non-auto-completing by construction, so there was never anything to turn off.

2. ASB does not need a new listener class. BatchedAzureServiceBusListener's receive loop is already non-blocking — it spawns its own Task.Run(listenForMessages) and hands each batch to ReceivedAsync without awaiting handler completion — and it carries no settlement policy of its own: CompleteAsync goes straight to ServiceBusReceiver.CompleteMessageAsync. It only looks like ack-at-receipt today because BufferedReceiver posts its complete block at receipt. Swap in NativeAckReceiver and the same class settles at handler completion, unchanged. So the issue's "third class or settlement-policy parameter?" question has a third answer for ASB: neither.

3. Pub/Sub is a different problem — it has no per-message settlement primitive at all, in any mode — and is now tracked on NativeAck endpoint mode for GCP Pub/Sub #4052. Nothing here touches it.

What was actually left for ASB is the two things below.

1. ISupportNativeScheduling on BatchedAzureServiceBusListener

Without it, MessageContext.ReScheduleAsync falls through to Storage.Inbox, which breaks #3708's storage-free criterion for the mode and throws outright on a host with no persistence.

Implemented the way the inline listener does it — stamp Envelope.ScheduledTime and re-publish through the requeue sender, which the outgoing mapper turns into ServiceBusMessage.ScheduledEnqueueTime, so the broker holds the message rather than Wolverine.

Two choices worth a reviewer's attention:

• It routes through the listener's existing _defer block rather than sending directly, so the original delivery is settled before the copy goes out. This is the one deliberate divergence from the inline listener, which does not settle the original. Within this class the precedent is unambiguous: the _defer block's own Azure Service Bus performance deep dive: PrefetchCount validation, session listener rework, settlement concurrency #3494 (AO8) comment records that leaving the original unsettled means the broker redelivers it at lock expiry, so every deferral costs a duplicate. (The inline listener's omission looks like a live defect since Azure Service Bus: inline processor leaves AutoCompleteMessages at the SDK default (true), so a failed dead-letter/defer can be auto-completed behind Wolverine's back #4018 turned off AutoCompleteMessages there — out of scope here, flagged separately.)
• NativeSchedulingEnabled is gated to EndpointMode.NativeAck alone. ReScheduleAsync prefers the listener over the pipeline channel, so answering true unconditionally would take the reschedule away from receivers that already own it: BufferedReceiver, which supplies an in-memory rescheduler, and DurableReceiver, whose envelope already has an inbox row that a republished copy under the same id would collide with on redelivery. That is exactly why RedisStreamListener opts out of Durable.

2. NativeAck + ASB sessions refused at bootstrap, Fatal

SessionSpecificListener.CompleteAsync throws NotSupportedException, and the accept loop scopes the session receiver to a single await using iteration — so under NativeAck the session lock is disposed before any handler has run. A lifetime mismatch, not a renewable timeout, so there is no tuning that rescues it.

Where the check lives, and why it cannot live in the Mode setter

RequireSessions() and ProcessInParallelWithNativeAcks() are both delayed configuration — each is an add(e => …) lambda applied in registration order inside Endpoint.Compile(). A guard in the Mode setter (or in supportsNativeAck, the only member that setter consults) therefore sees a different endpoint depending on which fluent call the user wrote first: sessions-first, it sees a session queue; native-acks-first, it sees a session-less one and waves the pair through. Same order-dependence class GH-3712 fixed for ProcessInline() + MaximumParallelMessages().

The check goes in ListenerConfigurationValidator, above its non-Inline early-out, so it runs over the final compiled state.

native_ack_with_sessions_4049.a_guard_in_the_mode_setter_would_be_ordering_dependent pins that premise as a test rather than a claim: the test subclass records Options.RequiresSession at the moment supportsNativeAck is read, and asserts true for one ordering and false for the other.

A defensive throw also guards the two listener-selection sites in AzureServiceBusTransport.Listening, which ask about RequiresSession ahead of the mode and would otherwise take the session branch in silence.

Relationship to #4057 — resolved: it merged, and the identical-copy strategy worked

This needs the general Endpoint.validateModeConfiguration() hook that #4057 added for Pulsar. #4057 was still open when this branch was written, so rather than invent a second mechanism, it carried byte-identical copies of #4057's three core changes — the hook on Endpoint.cs, its wiring in ListenerConfigurationValidator.cs, and the native_ack_mode_gate.cs additions — applied at the same locations from the same base commit, on the theory that git resolves identical changes from a common ancestor without a conflict.

#4057 has since merged, and that is exactly what happened. Merging the new main in produced no conflict in any of the three files, and this branch's copies simply merged away. The hook now comes from #4057 and appears exactly once.

The diff against main is now purely Azure Service Bus — the transport plus its two new test files. Nothing in src/Wolverine/ remains.

Scope — why not "Closes #4049"

Both halves the re-scoped issue asked for are done. But supportsNativeAck deliberately stays false on AzureServiceBusQueue and AzureServiceBusSubscription, so ASB still refuses ProcessInParallelWithNativeAcks() in the Mode setter today. Flipping it is the adoption follow-up — prefetch / MaximumMessagesToReceive sizing plus the compliance suite, gated on #4048 — and this PR is the prerequisite that has to be in place before that flip, so the issue should stay open until then.

The tests open only that one gate, through a test-only subclass. the_shipping_queue_still_refuses_native_acks_in_the_mode_setter pins the distinction explicitly, so a session-rejection test can never pass on the strength of the mode gate's own InvalidOperationException — a different exception type, from a different place, at a different time.

No user-facing docs, for the same reason: documenting a configuration nobody can currently write would be premature. Docs belong with the adoption PR.

3. A live regression on main that this surfaced (#4061 / GH-3709)

Not planned work — the end-to-end test found it, and it had to be fixed for ASB native acks to function at all.

Sending anything to an ASB endpoint that listens with native acks threw InvalidOperationException: "This sender has not been registered." on every batch. Mechanism:

1. GH-3709: send inline from NativeAck endpoints to close the interceptor loss window #4061 (merged into main while this branch was in flight) remapped EndpointMode.NativeAck to InlineSendingAgent in EndpointCollection.buildSendingAgent.
2. InlineSendingAgent is not an ISenderCallback, so CreateSendingAgent skips RegisterCallback — it only fires when sender is ISenderRequiresCallback && agent is ISenderCallback.
3. But this transport still chose a BatchedSender, because its gate asked only about EndpointMode.Inline.
4. BatchedSender.SendBatchAsync refuses to send without a callback.

Agent says inline, sender says batched. This is precisely the mismatch the existing GH-3826 comment a few lines above already documents for TenantedSender, reached through the mode rather than through tenancy — so the fix is the same one: NativeAck takes the inline sender.

It went unnoticed because native_ack_native_scheduling_4049.a_scheduled_retry_goes_back_through_the_broker is the first ASB test that both listens with native acks and publishes to the same queue.

⚠️ Redis is exposed the same way and is not fixed here. RedisStreamEndpoint.CreateSender has the identical Mode == EndpointMode.Inline gate, and Redis accepted NativeAck in #4056 — so any Redis stream that is both a native-ack listener and a send target hits this today. Pulsar's #4057 is worth checking too. Raised separately rather than patched here, because the recurring per-transport fix may be the wrong answer: a central guard (a shared "sends inline" predicate, or making CreateSendingAgent fail loudly on an agent/sender mismatch instead of at runtime) would stop this recurring for every future NativeAck adopter. That is a design call for the maintainer, not something to decide inside this PR.

Verification

• dotnet build wolverine.slnx -c Release -f net9.0 — the pinned form CI uses: clean, 0 warnings, 0 errors.
• Full CoreTests: 2581 total, 0 failed, 2 skipped (re-run after merging main, which changed the same validator).
• New: native_ack_with_sessions_4049 (12 tests) and native_ack_native_scheduling_4049 (3 tests, against the ASB emulator) — all green.

Red baselines — what was broken, and what failed

Every test that claims to prove new behaviour was run against a deliberately broken implementation. The two rejection branches were baselined independently, so neither can be passing on the other's account:

Break applied	What failed	What still passed
AzureServiceBusEndpoint.validateModeConfiguration() → yield break	the 4 ordering tests and the_host_refuses_to_start (5 failures)	the 3 listener-selection guard tests
AssertSessionsAreCompatibleWithMode() → body replaced with if (false)	the_listener_selection_guard_refuses_the_pair only (1 failure)	all 4 ordering tests + the_host_refuses_to_start
NativeSchedulingEnabled → false	the listener-shape test and a_scheduled_retry_goes_back_through_the_broker, the latter on attempts[1].Envelope should not be same as — i.e. the in-memory fallback re-executed the same Envelope instance, which is precisely the distinction the test exists to draw	buffered_mode_still_leaves_scheduling_to_its_receiver
republish with no scheduled enqueue time (ScheduledTime = null)	a_scheduled_retry_goes_back_through_the_broker, on the elapsed-time assertion	the other two

The sender fix in §3 has a natural before/after baseline rather than an injected one: a_scheduled_retry_goes_back_through_the_broker failed twice with "This sender has not been registered." before it and passes after.

A note on the local ASB runs. Another agent was running ASB suites against the same emulator throughout, restarting the container mid-run. That produced two rounds of spurious failures — 23 BufferedSendingAndReceivingCompliance tests failing at 0ms (fixture init), plus session_id_pinning, end_to_end_with_CloudEvents and end_to_end_with_conventional_routing timing out. Every one of them passes on an uncontended re-run, and I have not claimed any contended run as a result in either direction. Contention here only ever causes timeouts, never false passes, so the green results above stand.

The last row is what keeps the behavioural test from being tautological: it separates "a copy was republished" from "the broker held the copy for the requested delay". (It also turned up that the ScheduledTime stamp inside MoveToScheduledUntilAsync is redundant — ReScheduleAsync sets it first — so it is kept only for consistency with the inline listener and for direct callers.)

CI — 35 pass, 2 red, neither from this branch

CIAzureServiceBus passes (13m21s), along with CIAzureServiceBusLeader, CIAzureServiceBusRouting, CIPulsar and CIRabbitMQ. That is the authoritative full ASB suite, and it counts for more than the local runs described above, which were repeatedly contaminated by a concurrent suite restarting the emulator.

CIRedis — pre-existing breakage on main, and this PR's investigation identified its cause. CI bisect on main itself:

main commit	CIRedis
64cff90a9 (#4056, which added Redis NativeAck)	success
2ea4266b6 (#4061, GH-3709 "send inline from NativeAck endpoints")	failure

The failures are exactly and only the native-ack class — 154 passed, 4 failed, all four in Wolverine.Redis.Tests.native_ack_mode — and they are the same defect §3 fixes for Azure Service Bus. Reproduced locally, where the host log names it four times:

System.InvalidOperationException: This sender has not been registered.
   at Wolverine.Transports.Sending.BatchedSender.SendBatchAsync(...)

RedisStreamEndpoint.CreateSender (line 344) still gates its inline sender on Mode == EndpointMode.Inline, so a native-ack endpoint gets a BatchedSender under an InlineSendingAgent that never registers the callback. Redis native-ack messaging is non-functional on main today, and main's own CI has been red since #4061. Raised separately rather than fixed here — the recurring per-transport patch may be the wrong answer next to a central guard, and Pulsar (#4057) should be audited for the same gate. Worth checking before the next release.

CIMarten — conclusion is cancelled, not failure, with no step marked failed; gh pr checks renders that as fail. It hit the 20-minute cap. The in-job memory sampler from #3771 shows what happened and rules out the OOM hypothesis it was added to test:

[mem] 03:55:52 total=15989 used=2253 free=8745 available=13736 swap_used=0 | MartenTests=1160MB
##[error]The operation was canceled.

Byte-identical readings every 15 seconds for minutes, ~13.7 GB still available. A wedged suite, not memory exhaustion. CIMarten is green on main, and MartenTests.csproj has no reference to Wolverine.AzureServiceBus.

An earlier commit also saw CIPersistence go red; its SQL Server service container never came up on the runner (not ready after 120s ... TCP Provider, error: 35), so no test executed. It is green on main, and PersistenceTests passes locally here (105/105).

The diff against main is confined to src/Transports/Azure/, and neither MartenTests nor PersistenceTests nor Wolverine.Redis.Tests references Wolverine.AzureServiceBus.

🤖 Generated with Claude Code