Cryptic error "received a batch message with no data in it" when sending large messages in batch.

[arielpontes]

• Package Name: azure-servicebus
• Package Version: 7.12.3
• Operating System: macOS 14.5
• Python Version: 3.11.9

Describe the bug
Trying to write batches with large messages cause cryptic error.

To Reproduce
Steps to reproduce the behavior:

1. Run the following script:

import os

from azure.servicebus import ServiceBusClient, ServiceBusMessage

connstr = os.environ["QUEUE_SERVICE_CONNECTION_STRING"]


def print_exc(e):
    print(f"{type(e).__module__}.{type(e).__name__}: {e}\n")


with ServiceBusClient.from_connection_string(
    conn_str=connstr, logging_enable=True
) as client:
    with client.get_queue_sender("main-items") as sender:
        try:
            message = ServiceBusMessage("test message " * 999_999)
            sender.send_messages(message)
            print("Sent!")
        except Exception as e:
            print_exc(e)
        try:
            messages = [ServiceBusMessage("test message " * 999_999)]
            sender.send_messages(messages)
            print("Sent!")
        except Exception as e:
            print_exc(e)
        try:
            messages = [ServiceBusMessage("test message " * 99_999)]
            sender.send_messages(messages)
            print("Sent!")
        except Exception as e:
            print_exc(e)
        try:
            messages = [ServiceBusMessage("test message " * 9_999)]
            sender.send_messages(messages)
            print("Sent!")
        except Exception as e:
            print_exc(e)

Expected behavior
First two messages are not sent because they are too big, but the last two messages are sent.

What actually happens

I get this output:

azure.servicebus.exceptions.MessageSizeExceededError: The received message (delivery-id:92, size:2161566 bytes) exceeds the limit (2097152 bytes) currently allowed on the link. TrackingId:e01f5159-144c-4303-8612-6557134675f2_B33, SystemTracker:NoSystemTracker, Timestamp:2024-11-01T12:44:13 Error condition: amqp:link:message-size-exceeded.

azure.servicebus.exceptions.MessageSizeExceededError: ServiceBusMessageBatch has reached its size limit: 2097152

azure.servicebus.exceptions.ServiceBusError: The link 'G27:1046226:2160e792-ba44-478a-90fa-8c195899ae07' is force detached by the broker because publisher(link40917) received a batch message with no data in it. Detach origin: Publisher. Error condition: amqp:not-allowed.

Sent!

I would expect the third message to be sent, but if it's not I would at least expect the error to be more similar to the one I get for the third case:

azure.servicebus.exceptions.MessageSizeExceededError: ServiceBusMessageBatch has reached its size limit: 2097152

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[kashifkhan]

Hi @arielpontes ,

There are a couple of subtle things going in the examples that you have posted. The service allows a single message to be sent or a batch of messages to be sent ( which is a special collection called ServiceBusMessageBatch )

When using a premium service bus you are able to send payloads up to 100 MB however they can only be sent as single messages. Batching is not supported

Also batches have a max size of 1 mb in total on the premium tier, limits

Any time a message is sent as a list, its converted in to a batch, so the only one that technically should go across is the last one.

Ill try and repro the issue and see why you are getting the errors as shown.

[arielpontes]

Hi @kashifkhan , thanks for the quick reply! Indeed I'm using the premium tier, and we updated the max message size to 2048 kb. Let me see if I understand it correctly though: the premium tier doesn't support batching at all? Most of our messages shouldn't be that large, so I would expect batching in this case to improve efficiency quite significantly by reducing overhead. Am I missing something? Should I really just drop the batching altogether and send the messages one by one without worrying too much about reduced efficiency?

[kashifkhan]

Hi @arielpontes,

sorry my explanation was not clear

• Standard & Premium tiers support batching
• The total size of a batch must not exceed 256 kb when using Standard and 1 mb when using Premium.
• When using large messages batching is not supported, so you can't send a batch with a message of size 2mb or 100 mb for example

As long as your batches are less than 1 mb it should work in premium. We do have a bug in our error message that we have to address, but the functionality still works.

which is why the following happens in your code

• The first message fails as its larger than 2048 kb
• The second batch fails as its larger than 1 mb
• The third batch fails as its larger than 1 mb (we need to fix this error message but the functionality is proper)
• The fourth batch succeeds as its less than 1 mb

[arielpontes]

Thanks for the explanation! Things are getting clearer but I'm still trying to wrap my head around this issue. I tried to add the problematic message to a batch twice and got this error:

*** azure.servicebus.exceptions.MessageSizeExceededError: ServiceBusMessageBatch has reached its size limit: 2097152

Didn't you say the maximum batch size is 1mb? According to this error it says the maximum batch size is 2mb. Is this another buggy error message that I shouldn't take at face value? Or when you said the "third batch fails as its larger than 1 mb", did you actually mean the "third batch fails as it has a message that is larger than 1 mb"?

[kashifkhan]

no worries @arielpontes ,

That is an error message from the sdk that is buggy and when fixed should say

*** azure.servicebus.exceptions.MessageSizeExceededError: ServiceBusMessageBatch has reached its size limit: 102400 when someone is using a premium tier service bus and 25600 when someone is on standard. The SDK today is erroneously outputting the max allowed single message size here.

So if you set your queue to accept messages of size 50 mb for ex, that means you can send single messages (one at a time) up to 50 mb. Batches however are still limited to 1 mb in premium ( 256 kb in standard ) and this means the size of the entire batch has to be less than that (there is some overhead involved so the actual payload has to be a little smaller than that).

The third batch fails to send because the size of the batch has gone beyond 1 mb (and in this case because the single message itself is larger than 1 mb )

[ServiceBusMessage("test message " * 99_999)] --> in this case as its a list, its treated as a batch and here the message is larger than 1.24 mb it should fail. Now if you send this message as is, without a batch, itll go through.

case 1

A single message that is greater than 2048 kb and therefore fails as the limit on the queue is 2048 kb for a single message

message = ServiceBusMessage("test message " * 999_999)
sender.send_messages(message)

case 2

A single message that is greater than 2048 and is being treated as a batch because of the list.

The batch goes above the 1 mb limit here and the error message should tell you the right limit ... this is a bug we have to fix.

messages = [ServiceBusMessage("test message " * 999_999)]
sender.send_messages(messages)

case 3

A single message that is greater than 1 mb and is being treated as batch because of the list.

Even here the batch goes above the 1 mb limit and the error message should be the same as the one output from above but with the right batch limit ... same we need to fix this

messages = [ServiceBusMessage("test message " * 99_999)]
sender.send_messages(messages)

case 4
The message is well below 1 mb and its treated as a batch. The batch size here is well below 1 mb and you could send a few more messages in this batch. The message is sent fine

messages = [ServiceBusMessage("test message " * 9_999)]
sender.send_messages(messages)

[arielpontes]

Got it. Thank you for the detailed explanation @kashifkhan ! Just to summarize:

The ServiceBusMessageBatch class is buggy because it thinks the max batch size is the same as the max message size (2mb), but in fact it's 1mb. This causes multiple issues, such as:

• It allows us to add messages to a batch via .add_messages until the total batch size is 2mb. Instead, it should start erroring out as soon as the batch size reaches 1mb.
• When the oversized batch is finally sent, it causes confusing/cryptic error messages.

So basically I cannot use the pattern of adding messages to a batch within a try block until it raises an exception because it's full. I have to take care of managing the size of the batch myself until this bug is fixed. Is that accurate?

[kashifkhan]

@arielpontes yes that is correct. We are working on a fix for it asap so we can get it out in this months release. Sorry about the issues this is causing you, I will update the thread once the PR is merged and we are ready to release to pypi

[arielpontes]

Actually I can still use the same pattern, I just have to make sure to pass the max_size_in_bytes argument when creating the batch:

max_batch_size = 1024 * 1024  # 1mb
batch = await sender.create_message_batch(
    max_size_in_bytes=max_batch_size
)

I ran some tests and it worked. Just leaving this here in case it's useful for someone else :)

[kashifkhan]

@arielpontes we have now released a new version of SB which addresses this, so you no longer have to use the mitigation

[github-actions]

Hi @arielpontes. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.