HDDS-3432. Enable TestBlockDeletion test cases.

[lokeshj1703]

What changes were proposed in this pull request?

Enable block deletion test

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3432

How was this patch tested?

Enables a UT

[codecov-commenter]

Codecov Report

Merging #1121 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1121      +/-   ##
============================================
- Coverage     70.56%   70.55%   -0.02%     
- Complexity     9427     9437      +10     
============================================
  Files           965      967       +2     
  Lines         49063    49260     +197     
  Branches       4803     4825      +22     
============================================
+ Hits          34620    34754     +134     
- Misses        12137    12184      +47     
- Partials       2306     2322      +16     

Impacted Files	Coverage Δ	Complexity Δ
...ntainerLocationProtocolServerSideTranslatorPB.java	35.80% <0.00%> (-4.81%)	17.00% <0.00%> (-2.00%)
...ntainerLocationProtocolClientSideTranslatorPB.java	39.06% <0.00%> (-3.73%)	20.00% <0.00%> (-2.00%)
...apache/hadoop/hdds/scm/block/BlockManagerImpl.java	68.46% <0.00%> (-3.61%)	19.00% <0.00%> (+1.00%)	⬇️
...adoop/hdds/scm/server/SCMClientProtocolServer.java	47.47% <0.00%> (-3.51%)	23.00% <0.00%> (-4.00%)
.../apache/hadoop/hdds/scm/node/NodeStateManager.java	75.00% <0.00%> (-3.05%)	42.00% <0.00%> (-1.00%)
...doop/ozone/om/exceptions/OMNotLeaderException.java	48.48% <0.00%> (-3.04%)	3.00% <0.00%> (-1.00%)
...ent/algorithms/SCMContainerPlacementRackAware.java	76.69% <0.00%> (-3.01%)	31.00% <0.00%> (-2.00%)
...r/common/states/datanode/RunningDatanodeState.java	79.22% <0.00%> (-2.60%)	18.00% <0.00%> (-1.00%)
...hadoop/ozone/om/protocolPB/Hadoop3OmTransport.java	74.00% <0.00%> (-2.55%)	18.00% <0.00%> (+1.00%)	⬇️
...e/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java	67.52% <0.00%> (-2.53%)	28.00% <0.00%> (ø%)
... and 33 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f8fcc47...2e5e84f. Read the comment docs.

[lokeshj1703]

Ran a workflow on the branch.
https://github.com/lokeshj1703/hadoop-ozone/actions?page=1&query=workflow%3Ait-client
The test failed 5 times out of 100.

[adoroszlai]

Thanks @lokeshj1703 for working on this. 3/20 runs failed with two different problems:

1. assertion failure at TestBlockDeletion.verifyPendingDeleteEvent(TestBlockDeletion.java:275)
2. timeout at TestBlockDeletion.testBlockDeletion(TestBlockDeletion.java:200)

https://github.com/adoroszlai/hadoop-ozone/runs/808704334

[lokeshj1703]

Thanks @adoroszlai for verifying! I have pushed a commit which fixes the timeout issue. For the time being I have ignored the verifyPendingDeleteEvent check. I will need to think of another way for that test.

FYI - I had enabled a workflow for this test at https://github.com/lokeshj1703/hadoop-ozone/actions?query=workflow%3ATestBlockDeletion2. I am not sure if it takes the latest changes from the branch though.

[adoroszlai]

Thanks @lokeshj1703 for updating the patch. 1/20 runs still failed with timeout: https://github.com/adoroszlai/hadoop-ozone/runs/814060939

[lokeshj1703]

@adoroszlai Thanks for verifying! The test failed in the 20th iteration and it failed because of appendEntries timeout.
But I can see other failures like https://github.com/lokeshj1703/hadoop-ozone/runs/812429922?check_suite_focus=true in my workflow. I am currently working on this failure. Will update once I have all green runs except the appendEntries timeout issue.

[lokeshj1703]

@adoroszlai The workflow is now green. There were 2 failures in around 70 runs. One where cluster start timed out at waitForClusterToBeReady and another with appendEntriesTimeout.
https://github.com/lokeshj1703/hadoop-ozone/actions?query=workflow%3ATestBlockDeletion2

[adoroszlai]

Thanks @lokeshj1703 for updating the patch. I still see 1/20 failure due to timeout at TestBlockDeletion.testBlockDeletion(TestBlockDeletion.java:174).

https://github.com/adoroszlai/hadoop-ozone/runs/844624809

[lokeshj1703]

@adoroszlai Thanks for verifying! I see appendEntries timeout in the particular run. The blocks creation in all the datanodes timed out in the run. The failure occurs in iteration 12.

[elek]

/pending "I still see 1/20 failure due to timeout at TestBlockDeletion.testBlockDeletion(TestBlockDeletion.java:174)."

[github-actions]

Marking this issue as un-mergeable as requested.

Please use /ready comment when it's resolved.

"I still see 1/20 failure due to timeout at TestBlockDeletion.testBlockDeletion(TestBlockDeletion.java:174)."

[github-actions]

Thank you very much for the patch. I am closing this PR temporarily as there was no activity recently and it is waiting for response from its author.

It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.

It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.

If you need ANY help to finish this PR, please contact the community on the mailing list or the slack channel."