[SERVICE BUS] Consider adding optional drain to batch receive

[m-redding]

Library name

Azure.Messaging.ServiceBus

Please describe the feature.

When using the ServiceBusSessionReceiver or ServiceBusReceiver to receive a batch of messages, messages may be out of order when send and receive calls are happening simultaneously. This happens only when you have a sequence like the following: receive call ends because there are no more messages, more messages come in before the next receive call is started. This causes the receiver to release those in between messages since the excess credits are not being drained by default. When released, the service places them behind any additional messages that come in after the start of the call to receive.

To remedy this, consider adding an additional option to drain the excess receive credit. This option would hurt performance when used with concurrency, since credits could have been taken by other calls, so it must be an optional parameter set by the user. This option would presumably be the most useful in the ServiceBusSessionReceiver, where the ordering of batches is important.

[JoshLove-msft]

This has come up recently with a customer. I think we should add this optional parameter or otherwise address in some way.

[JoshLove-msft]

I would propose that we prototype the change to always drain when receiving and do some performance testing to see whether it has a noticeable impact on throughput. If the change is negligible we could ship it with drain. This behavior wouldn't be part of the public contract so we could always revert it if we needed to, or even add a flag in the future that says "do not drain" that can be set by anyone that is negatively impacted by the change.

[JoshLove-msft]

After doing a quick prototype and perf testing, drain substantially reduces throughput (~40% drop). So we would need to make this an opt-in setting, probably could do it in the ServiceBusReceiverOptions/ServiceBusSessionReceiverOptions (and the processor equivalents).