fix(rabbitmq): ack only the completed delivery, not every lower tag (GH-3492)

[jeremydmiller]

Draft — this change is incomplete on its own and must not be merged as-is. Split out of #3672
so the measured RO6 win there is not blocked on it.

The bug being fixed

RabbitMqListener.CompleteAsync calls BasicAckAsync(deliveryTag, multiple: true). In AMQP,
multiple: true means "ack this delivery tag and every lower unacked tag on this channel".

Completions finish out of order under any concurrent listener — BufferedInMemory or Durable
with MaxDegreeOfParallelism above 1. So completing tag 10 silently acks tags 1-9 while those
messages are still in the handler pipeline. If the process crashes in that window, the broker
considers them delivered and they are gone. That is a real at-least-once hole, and it is invisible
in normal operation because the messages usually do complete a moment later.

The one-line fix is multiple: false.

Why this is a draft: the sweep is load-bearing

CI caught what my local baseline did not. The cumulative ack is currently the only thing
settling deliveries that some code paths never acknowledge at all. Removing it makes those leak.

Measured, after a full Wolverine.RabbitMQ.Tests run against a freshly deleted queue:

Build	quorum1 after the full suite
origin/main	0 messages
this branch	1 message ready (leaked)

The consequence in the test suite: that leaked message is redelivered into a later run and shows up
as the last MessageSucceeded record in a tracked session, failing
quorum_queue_compliance.will_requeue_and_increment_attempts. The CI trace shows the intruder
arriving 3,010 ms into the test — a scheduled retry from a previous test method — with a different
message id than the one under test.

The consequence in production is worse than a flaky test: leaked deliveries sit unacked until the
channel closes, then get redelivered and reprocessed as duplicates.

Evidence trail

• CIRabbitMQ failed on perf(rabbitmq): per-endpoint consumer dispatch concurrency (GH-3492) #3672 twice, on fresh containers, with this change included —
  will_requeue_and_increment_attempts failing 2-of-2 retry attempts each time.
• perf(sqs): batch message deletions, and chunk outgoing batches on the request size limit (GH-3493) #3673 and fix(asb): de-quadratic session listeners, surface MaxConcurrentCalls, settle the original on a batched defer (GH-3494) #3674, branched off the same base commit with zero RabbitMQ changes, kept CIRabbitMQ
  green. Same CI, same clean containers, same full suite.
• Locally the test passes against a clean queue and fails against a dirty one on both branches,
  which is exactly why my local baselining missed this and briefly convinced me it was pre-existing
  flakiness.

What has to happen before this can merge

Find and settle the leaking path, rather than keeping the sweep to paper over it.

Prime suspect: RabbitMqInteropFriendlyCallback.MoveToErrorsAsync posts a copy of the envelope to
the dead-letter queue and never acks or nacks the original delivery — unlike
RabbitMqChannelCallback.moveToErrorQueueAsync, which does nack. There may be more than one such
path; the dead-letter route with DisableDeadLetterQueueing() (where NativeDeadLetterQueueEnabled
is false) is worth auditing too.

Note the leak does not reproduce when the quorum compliance class runs alone — only under the full
suite — so whatever it is, it is timing or ordering sensitive.

Reproduction

docker exec wolverine-rabbitmq-1 rabbitmqctl delete_queue quorum1
dotnet test src/Transports/RabbitMQ/Wolverine.RabbitMQ.Tests/Wolverine.RabbitMQ.Tests.csproj --framework net9.0
docker exec wolverine-rabbitmq-1 rabbitmqctl list_queues name messages messages_ready messages_unacknowledged

quorum1 should end at 0. On this branch it ends at 1.

🤖 Generated with Claude Code

[jeremydmiller]

Both sides of the leak measurement are now complete — full Wolverine.RabbitMQ.Tests run on the same box, same broker, quorum1 deleted before each:

Build	Suite result	quorum1 after
origin/main	480 passed / 3 failed (9m51s)	0 messages
this branch	483 passed / 3 failed (10m41s)	1 message ready

(The 3 extra tests on this branch are unrelated unit tests that were present when the measurement ran. Both sides had the same 3 pre-existing suite failures, so the leak — not a new test failure — is the differentiator locally.)

This is the reproduction to work against: whatever settle path fails to acknowledge its delivery shows up as exactly one stranded message, and only under the full suite, not when the quorum compliance class runs alone.

[jeremydmiller]

Superseded by #3737, which lands the same flip plus the settle fixes that make it safe. See that PR for why this one leaked into quorum1: the genuinely unsettled path turned out to be WorkerQueueMessageConsumer's un-mappable-message branch, not the interop DLQ callback.