Ack only the completed delivery, not every lower tag (GH-3706)

[jeremydmiller]

Closes #3706. Supersedes the draft #3675, which stopped at the flip itself.

The change

BasicAckAsync(tag, multiple: true) → multiple: false.

Cumulative acks are only correct when completions happen in delivery order, and they do not — out-of-order completion already exists today with ConsumerDispatchConcurrency > 1, and it becomes the normal case for the planned native-ack parallel endpoint mode. Acking tag N silently swept up every lower unacked tag on the channel, including deliveries whose handlers were still running; a crash at that moment is silent message loss.

Why the GH-3492 attempt leaked, and what actually leaked

The previous attempt measured throughput-neutral but leaked one message into quorum1, because the cumulative sweep was covering for settle paths that never acknowledged a delivery of their own. I audited every use of RabbitMqEnvelope for a path where neither ack nor nack fires and found one genuine orphan — plus one that the issue flags which turns out not to be:

• WorkerQueueMessageConsumer's un-mappable-message branch — dead-letters the envelope through the receiver and returns without touching the delivery at all. This is the real leak, and it isn't the one the issue names.
• RabbitMqInteropFriendlyCallback.MoveToErrorsAsync — the issue's candidate. It genuinely doesn't settle, but on the normal error-handling path that survives, because MoveToErrorQueue.ExecuteAsync calls lifecycle.CompleteAsync() immediately after MoveToDeadLetterQueueAsync, and NoHandlerContinuation does the same. I verified this empirically — disabling my settle here and re-running the repro test still passed — so the issue's "the cumulative sweep from later acks is what currently reclaims those orphaned deliveries" isn't accurate for this path. Settled anyway, so the delivery doesn't depend on a later step in a different layer for the only thing that ever reclaims it.

Both settle with an ack, not a nack. The sibling RabbitMqChannelCallback.moveToErrorQueueAsync nacks with requeue: false precisely so the broker's own x-dead-letter-exchange routes the original — that's native dead lettering and it sends no copy of its own. These two paths have already taken responsibility by writing the message elsewhere, so a nack would either be discarded (InteropFriendly removes the DLX argument from the queue) or, under UseEnhancedDeadLettering against a Native-mode queue that still has a DLX, put a second copy in the dead letter queue.

CanSettle stays. Its doc comment justified itself partly on cumulative acks making a replayed stale tag catastrophic; per-message acks narrow that blast radius to one wrong message but don't make settling on a replaced channel correct, so the comment is corrected rather than the guard removed.

Not resurrected: ack coalescing/batching, prototyped and rejected at −10% (ledger row 216) — basic.ack is a fire-and-forget frame, not an RPC, so batching saves no round trips.

Tests

Bug_3706_dead_letter_paths_settle_the_original_delivery asserts that a delivery which has been through a dead-letter path leaves the source queue empty after the channel closes — the broker requeues anything still unacked when a channel goes away, which is exactly how the GH-3492 leak manifested — and that exactly one copy lands in the dead letter queue rather than two. Covers the interop-friendly path and Native mode as the control.

Full local suite (the acceptance criterion on the issue)

Wolverine.RabbitMQ.Tests, 10m26s: 488 passed / 2 failed. Both failures investigated rather than assumed:

• multi_tenancy_through_virtual_hosts.send_message_to_a_specific_tenant — fails identically on unmodified main (same test, same line 171). Pre-existing; it's a tracked-session condition gap, the response is visibly Sent in the activity table before the session gives up.
• end_to_end.use_direct_exchange_with_binding_key — passes in isolation on both main and this branch, and the whole end_to_end class passes 20/20 here. Cross-class order dependence, matching the chronic first-round pattern tracked in CIRabbitMQ: three tests fail round one on every run and are masked by the flaky-retry harness #3726.

No message leaked into a quorum queue — the specific failure the previous attempt hit.

Full wolverine.slnx Release build clean.

Docs

New "Acknowledgements are per message" section in the RabbitMQ performance guide.

Unblocks the native-ack parallel endpoint mode issue.

🤖 Generated with Claude Code