HDDS-8880. Intermittent fork timeout in TestOMRatisSnapshots

[xBis7]

What changes were proposed in this pull request?

This patch fixes issues with three methods running under TestOMRatisSnapshots. This class was 30% of the time resulting in a timeout due to the configuration setup.

The issue comes from the config key ozone.om.ratis.snapshot.auto.trigger.threshold which is referring to the number of unappended logs before triggering a snapshot installation. All tests under this class depend on frequent Ratis snapshot installations.

#4770 Increased the number of writes performed by testInstallSnapshot() which in combination with the frequent Ratis snapshot installations led 30% of runs to a system freeze that caused a timeout. To fix the issue, only testInstallSnapshot() runs configured with a higher threshold number.

Furthermore, testInstallIncrementalSnapshot and testInstallIncrementalSnapshotWithFailure are installing a Ratis snapshot on the follower and then checking that the follower's key table has the keys from the Ratis snapshot. Sometimes this check is taking place sooner than expected making the methods flaky. By adding a wait check on reading the key table, the issue is fixed.

There is an existing issue with testInstallIncrementalSnapshotWithFailure, that hasn't been fixed. At some point we are expecting that there is a new checkpoint generated in the leader OM and we check the metrics to see if they are updated accordingly. Rarely, this doesn't happen and there is a failure. I added the @Flaky annotation above the method until I address the issue in HDDS-8876.

This class is still flaky but I'm creating this PR to remove the @Disabled annotation and get this test class running along with the rest of the CI.

Should we add

@Flaky("HDDS-8876")

above the class as well?

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8880

https://issues.apache.org/jira/browse/HDDS-8952

How was this patch tested?

CI on my fork: https://github.com/xBis7/ozone/actions/runs/5452838798

CI running TestOMRatisSnapshot in 100 iterations

https://github.com/xBis7/ozone/actions/runs/5447212761

https://github.com/xBis7/ozone/actions/runs/5447866879

https://github.com/xBis7/ozone/actions/runs/5447868491

We are getting a fork timeout in 1/100 or 2/100 runs.

https://github.com/xBis7/ozone/actions/runs/5447868491/jobs/9910384871#step:5:4472

[xBis7]

@adoroszlai @GeorgeJahad Can you please take a look at this PR?

[adoroszlai]

Thanks @xBis7 for the patch and extensive testing. 👍

[xBis7]

@adoroszlai Thanks for the review.

[xBis7]

@adoroszlai Should we mark the class as flaky? Or adding the annotation to a method makes the whole class run under the flaky CI?

[adoroszlai]

adding the annotation to a method makes the whole class run under the flaky CI?

Only the methods tagged as @Flaky are run in the flaky split.

Should we mark the class as flaky?

The class should be tagged if all (or most) methods may fail intermittently (and the problem is such that repeated attempts may succeed).

[xBis7]

@adoroszlai Thanks for all the help and the info.

The class should be tagged if all (or most) methods may fail intermittently (and the problem is such that repeated attempts may succeed).

That's not the case here, we might get 1 or 2 failures every 100 runs. I'll try to address that in HDDS-8876.