HDDS-6890. EC: Fix potential wrong replica read with over-replicated container.

[guihecheng]

Co-authored-by: Uma Maheswara Rao G [email protected]

What changes were proposed in this pull request?

Fix potential wrong replica read with over-replicated container.
The problem is due to that an EC pipeline with multiple DNs is returned and used with the first DN picked while refreshing pipeline on read error(Read error is due to concurrent redundant EC container replica deletion, e.g. EC 10+4 with 18 replicas),
but the right behavior should be pick the DN with the same replicaIndex and build a standalone pipeline for retry.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6890

How was this patch tested?

To be verified manually.

[guihecheng]

cc @umamaheswararao @sodonnel for review.

[sodonnel]

I think this change looks good. I think it will fix the issue.

A test to reproduce the problem is likely too difficult, but I'd like to see a test that executes the new refresh wrapper.

Do you think we could create a protected method called something like:

Function<BlockID, Pipeline> ecPipelineRefreshFunction(BlockId blockID, int replicaIndex, Function<BlockID, Pipeline> refreshFunction) {
...
}

Then we can create a simple unit test that creates a simple refresh function that returns an hardcoded ECPipeline object, that is used by our new wrapper, and validate the correct node is selected based on the index?

[guihecheng]

I think this change looks good. I think it will fix the issue.

A test to reproduce the problem is likely too difficult, but I'd like to see a test that executes the new refresh wrapper.

Do you think we could create a protected method called something like:

Function<BlockID, Pipeline> ecPipelineRefreshFunction(BlockId blockID, int replicaIndex, Function<BlockID, Pipeline> refreshFunction) {
...
}

Then we can create a simple unit test that creates a simple refresh function that returns an hardcoded ECPipeline object, that is used by our new wrapper, and validate the correct node is selected based on the index?

Oh, that's a good idea to cover this small function with test, Thanks, I'll update soon.

[sodonnel]

LGTM. Thanks for adding the unit test.