batch_send does not account for SQS 256kb batch size limit

[correnarsprowls]

Hello!

I am having issues moving messages from one queue to another. Sometimes messages are nearly as large as the 256kb batch limit enforced by AWS SQS.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html

This limit causes this failure to occur in batch_send : Aws::SQS::Errors::BatchRequestTooLong: Batch requests cannot be longer than 262144 bytes. You have sent 1024000 bytes.

When this error occurs the messages are still considered "in flight" and will only become available again after a timeout. This makes it difficult to retry the failed batches in smaller sizes later because the queue will appear to be empty.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

[cjlarose]

I could use some help understanding a little bit more of the context.

Did you write a Shoryuken worker whose job it is to republish messages to a new queue? How is that worker configured (what shoryuken_options did you use)?

Something like this?

class MoveMesasgeWorker < Shoryuken::Worker
  shoryuken_options batch: true
  
  def perform(sqs_messages, _)
    Shoryuken.sqs_client.send_message_batch queue_url: queue_url,
                                            entries: sqs_messages
  end
end

[correnarsprowls]

We're actually manually starting it through our own CLI. Behind the scenes it's spinning up a single ECS task and then running the shoryuken mv CLI command

    def self.mv(from_name:, to_name:)
     _number_of_slices(from_name).times do
       `bundle exec shoryuken sqs mv #{from_name} #{to_name} -n #{MESSAGES_PER_MV}`
     end

     OpenStruct.new(:success? => true)
    end

Number of slices is the approximate_number_of_messages / MESSAGES_PER_MV

We tried capturing the output and if the Aws::SQS::Errors::BatchRequestTooLong error was present we tried running this again with the messages per move being less than the 10 message batches Shoryuken defaults to, but at that point the messages are in flight and the queue appears to be empty.

[cjlarose]

Thanks for that clarification! It all makes sense. It definitely seems like a legitimate bug with shoryuken mv and probably also shoryuken requeue since they both use batch_send which you mentioned as being the culprit.

I think the ideal solution would be for batch_send to partition the messages up into batches that were each smaller than the 256KB limit.

I saw you opened #661. Is that ready for review?

[correnarsprowls]

I accidentally opened that pr that was meant for our fork. #664 Here's the real one! Thank you!

[cjlarose]

Fixed by #664

[cjlarose]

A fix for this issue was deployed as part of Shoryuken 5.2.1