SQS QueuePoller sometimes returns dupe messages, raises BatchEntryIdsNotDistinct

[jdelStrother]

We use Aws::SQS::QueuePoller like so:

poller = Aws::SQS::QueuePoller.new(queue_url, idle_timeout: 0, wait_time_seconds: 1)
poller.poll(max_number_of_messages: 10) do |batch|
  do_work(batch)
end

Very rarely - say, every 100k requests - batch contains duplicate messages, with the same message_id & body, but different receipt_handles.

When QueuePoller tries to clean up these messages (after do_work is complete), it calls delete_message_batch which raises Aws::SQS::Errors::BatchEntryIdsNotDistinct.

I've not discovered this behaviour documented anywhere (that SQS might occasionally return duplicate messages when calling ReceiveMessage). It's presumably not considered a bug in SQS itself (..?) so should the SDK fix it by de-duping in Client#receive_message or QueuePoller#receive_messages ? Or is this considered an end-user problem to deal with?

[alextwoods]

Hey - I've created an issue with this: #2913 to track investigation.

Assuming you're not using a FIFO queue, then this is expected behavior - SQS only guarantees "deliver at least once" behavior and application codes does need to be able to handle duplicate messages. However, I think that duplicate messages within one batch should potentially be handled by the SDK's QueuePoller code rather than the end user.

[mullermp]

#2918

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.