Fixing flaky test in ClusterManagerDisruptionIT

[jaideep-m]

Description

Fixes Flakiness of IT - testIsolateClusterManagerAndVerifyClusterStateConsensus in ClusterManagerDisruptionIT

The current test verifies

1. Cluster State Updates:

• After a network partition is healed, the cluster will attempt to reconcile the states of all nodes.
• However, the process of updating the cluster state is asynchronous and depends on various factors.

2. Failed Updates:

• The assertion clusterStateStats.getUpdateFailed() > 0 assumes that there will always be failed cluster state updates on the previously isolated node.
• This assumption may not always hold true, especially if: a) The cluster reconciles quickly without conflicts. b) The timing of the check happens after successful reconciliation.

The proposed approach is better for the following reasons:

1. Broader Coverage:

• It checks for any kind of cluster state activity, not just failed updates.
• This can catch scenarios where the cluster state changed successfully or where time was spent on updates without necessarily failing.

2. Reduced Flakiness:

• The original test might fail if the cluster manages to reconcile without any failed updates, which could happen in some scenarios.
• The new approach will pass if there's any indication of cluster state activity, reducing false negatives.

3. Enhanced Assertion Logic:

• Previously: Only checked for failed cluster state updates.
• Now: Verifies any cluster state activity (failed updates, successful updates, or time spent on updates).

4. Improved Logging:

• Added detailed logging of cluster state statistics for better diagnostics and debugging.

5. Timeout Adjustment:

• Implemented assertBusy with a 30-second timeout to allow sufficient time for cluster state changes to occur and be detected.

Related Issues

Resolves #[12095]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 99001ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?