Scaling Microservice Out - Multiple Instances of Same Message Processed

[douglaswaugh]

Is there an existing issue for this?

• I have searched the existing issues

Description

We want to scale out one of our services but we're getting a primary key violation about half the time when handling Volo.Abp.Users.User.Created messages when we do so.

Reproduction Steps

• Make a service which creates user entities
• Configure that service to use RabbitMQ as it's distributed bus with a direct exchange
• Make another service with a IDistributedEventHandler<EntityCreatedEto> to handle the Volo.Abp.Users.User.Created message and store the user in the services own users table
• Configure the consuming service with Prefetch = 1
• Scale the instances of the consuming service out to multiple instances (I used 5 instances in AKS) with the same client name
• Run a load test to generate sufficient load on the system. I'm creating around 500 users in 20 minutes.

Expected behavior

I would expect each message to be generally consumed once. Obviously messaging system are designed to work in an at-least-once manner in order to prevent lost messages, but the scale of the duplicates is much more than I would expect.

Actual behavior

The user gets inserted in to the consuming services database, but then further errors are logged when the same user is attempted to be inserted in to the database within a few milliseconds for the successful insert, but failing with a primary key violation. The insert is then tried an additional two times, potentially through some sort of retry mechanism?

As far as I can tell RabbitMQ is configured correctly and should be providing the messages in a round robin fashion. I don't know if there's some reason that ABP might take a message but not immediately set it as unacked which might mean that another instance has the opportunity to pick the message up?

Regression?

No response

Known Workarounds

I know that we should be deduplicating messages anyway, but this issue has come up before we've got that far and there are far more duplicates than I would expect which I think indicates a problem.

Version

8.0.1

User Interface

Common (Default)

Database Provider

EF Core (Default)

Tiered or separate authentication server

Separate Auth Server

Operation System

Linux

Other information

No response