HDDS-4013. FLAKY-UT: TestWatchForCommit#testWatchForCommitForGroupMismatchException

[adoroszlai]

What changes were proposed in this pull request?

testWatchForCommitForGroupMismatchException may fail with Group ... not found in waitForPipelineClose. The method destroys pipelines via regular SCM -> datanodes path, but then also calls removeGroup for each member manually.

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/ClosePipelineCommandHandler.java

Line 74 in f30aba7

server.removeGroup(pipelineID.getProtobuf());

ozone/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestHelper.java

Line 242 in f30aba7

server.removeGroup(pipeline.getId().getProtobuf());

Only one of these can succeed, the group cannot be removed twice. If test code wins, we see this error in the output:

2020-12-11 11:59:14,841 [Command processor thread] ERROR commandhandler.ClosePipelineCommandHandler (ClosePipelineCommandHandler.java:handle(78)) - Can't close pipeline PipelineID=82ba59fa-dd35-4d4e-a61e-59b281828b58
java.io.IOException: 675d35ba-bb57-4ae2-b121-2cd47ec358f4: Group group-59B281828B58 not found.
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:774)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ClosePipelineCommandHandler.handle(ClosePipelineCommandHandler.java:74)

But if ClosePipelineCommandHandler is first to remove, the test fails:

testWatchForCommitForGroupMismatchException(org.apache.hadoop.ozone.client.rpc.TestWatchForCommit)  Time elapsed: 39.399 s  <<< ERROR!
java.io.IOException: 14058016-3fcf-4920-991e-a5361899d5d7: Group group-59B281828B58 not found.
	at org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.removeGroup(XceiverServerRatis.java:774)
	at org.apache.hadoop.ozone.container.TestHelper.waitForPipelineClose(TestHelper.java:242)
	at org.apache.hadoop.ozone.client.rpc.TestWatchForCommit.testWatchForCommitForGroupMismatchException(TestWatchForCommit.java:353)

https://issues.apache.org/jira/browse/HDDS-4013

How was this patch tested?

Ran TestWatchForCommit 50x - never failed, only timed out once:
https://github.com/adoroszlai/hadoop-ozone/runs/1544890660

Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/418502626

[ayushtkn]

Thanx @adoroszlai for the fix, I am able to repro this, changes seems fair enough.
Does it makes sense to removeGroup explicitly in case server.isExist(pipelineId) is true, rather than waiting? Might speed up test? We can ignore the exception from removeGroup in the wait and retry to counter the race condition

[adoroszlai]

Thanks @ayushtkn for taking a look.

I'd rather avoid such "manual intervention" and let the test exercise production code paths.

I don't think keeping removeGroup would result in much speedup. If SCM and datanodes take too much time to execute regular flow, we could probably tweak executor intervals, etc.

It might be possible to convert it to a unit test with mocks, resulting in much bigger speedup by avoiding the mini cluster.

[ayushtkn]

makes better to stay near the prod code. Thanx

[adoroszlai]

Thanks @ayushtkn for the review.