SQS QueuePoller sometimes returns dupe messages, raises BatchEntryIdsNotDistinct

[alextwoods]

Discussed in #2908

^{Originally posted by jdelStrother September 6, 2023}
We use Aws::SQS::QueuePoller like so:

poller = Aws::SQS::QueuePoller.new(queue_url, idle_timeout: 0, wait_time_seconds: 1)
poller.poll(max_number_of_messages: 10) do |batch|
  do_work(batch)
end

Very rarely - say, every 100k requests - batch contains duplicate messages, with the same message_id & body, but different receipt_handles.

When QueuePoller tries to clean up these messages (after do_work is complete), it calls delete_message_batch which raises Aws::SQS::Errors::BatchEntryIdsNotDistinct.

I've not discovered this behaviour documented anywhere (that SQS might occasionally return duplicate messages when calling ReceiveMessage). It's presumably not considered a bug in SQS itself (..?) so should the SDK fix it by de-duping in Client#receive_message or QueuePoller#receive_messages ? Or is this considered an end-user problem to deal with?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.