[Meta] Fix random test failures

[anasalkouz]

PRs were blocked by transient gradle check errors multiple times. Provide a plan to stabilize the tests.

• [BUG] Intermittent test failure - Snapshot and Restore with repository-s3 #4069
• org.opensearch.upgrades.FullClusterRestartIT.testEmptyShard #2434
• [BUG] org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout #2242
• [BUG] ShardIndexingPressureConcurrentExecutionTests.testCoordinatingPrimaryThreadedUpdateToShardLimits assertion error #2241
• [BUG] o.o.search.SearchCancellationIT.testCancellationDuringFetchPhaseUsingRequestParameter #2311
• [BUG] org.opensearch.gateway.QuorumGatewayIT > testQuorumRecovery FAILED #1908
• [BUG] node drop on o.o.cluster.routing.allocation.decider.MockDiskUsagesIT.testRerouteOccursOnDiskPassingHighWatermark #1907
• [CI] oldClusterTest fails intermittently #1793
• [BUG] ClusterRerouteIT.testDelayWithALargeAmountOfShards (random test failure) #1561
• [BUG] SharedClusterSnapshotRestoreIT.testSnapshotCanceledOnRemovedShard (Random test failure) #1562
• [BUG] MinimumMasterNodesIT.testThreeNodesNoMasterBlock (random test failure) #1563
• [BUG] InstallPluginCommandTests.testOfficialPluginSnapshot (Random Test Failure) #1564
• [BUG] StableClusterManagerDisruptionIT.testStaleClusterManagerNotHijackingMajority (Random Test Failure) #1565
• [BUG] cluster.routing.PrimaryAllocationIT.testPrimaryReplicaResyncFailed intermittently fails #1722
• [CI] o.o.gateway.RecoveryFromGatewayIT.testReuseInFileBasedPeerRecovery #1746
• [CI] o.o..search.aggregations.metrics.TDigestStateTests.testMoreThan4BValues failure #1756
• [CI] oldClusterTest fails intermittently #1793
• [BUG] Cluster manager bootstrap takes time causing intermittent failures in integration tests (o.o.c.ClusterHealthIT.testHealthOnClusterManagerFailover) #1828
• [BUG] Transport NIO test intermittently failing for large responses #1844
• [BUG] Netty Transport test failing with large responses #1847
• [BUG] :distribution:packages:rpm:checkExtraction intermittent failure #1852
• [BUG] Timeout on org.opensearch.cluster.routing.MovePrimaryFirstTests.testClusterGreenAfterPartialRelocation #1957
• [BUG] org.opensearch.common.network.InetAddressesTests.testForStringIPv6WithScopeIdInput #1887
• [BUG] :test:fixtures:s3-fixture:composeUp (random test failure) #1566
• [BUG] FsHealthServiceTests.testFailsHealthOnHungIOBeyondHealthyTimeout (Random test Failure) #1567
• [CI] o.o.cluster.ClusterHealthIT.testHealthOnMasterFailover test failure #1693
• [CI] o.o.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks multiple failures #1703
• [CI] o.o.transport.netty4.OpenSearchLoggingHandlerIT fails w/ stack overflow #1767
• [CI] :test:fixtures:s3-fixture:composeUp fails due to HTTP connection issue #1806
• CompositeAggregatorTests.testWithKeywordLongAndMissingBucket test failure #2012
• [BUG] org.opensearch.common.settings.ScopedSettingsTests/testValidate flaky #5222
• MasterServiceTests.testThrottlingForMultipleTaskTypes #5958
• [BUG] Flaky test org.opensearch.remotestore.RemoteStoreIT.testRemoteStoreRestoreFromRemoteSegmentStore #6248
• [BUG] org.opensearch.index.translog.LocalTranslogTests.testWithRandomException is flaky #6571
• [BUG] org.opensearch.snapshots.SearchableSnapshotIT.testCacheFilesAreClosedAfterUse is flaky #6686
• [BUG] org.opensearch.client.indices.GetIndexTemplatesResponseTests.testParsingFromOpenSearchResponse is flaky #6739
• [BUG] org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure is flaky #7703

[andrross]

I did a quick experiment overnight on my dev machine where I ran the internalClusterTest all night in a loop:

for i in $(seq 0 1000) ; do echo "Iteration: $i" && ./gradlew ':server:internalClusterTest' >> test-output.txt 2>&1 ; done

Results:

$ egrep 'BUILD (SUCCESSFUL|FAILED)' test-output.txt | wc -l
152
$ egrep 'BUILD FAILED' test-output.txt | wc -l
3

$ egrep '^REPRODUCE' test-output.txt | less -S | uniq
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate" -Dtests.seed=7B8B067879F3C91F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Brazil/West -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting" -Dtests.seed=9F8306D99E2C2EF1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=id -Dtests.timezone=Asia/Aqtau -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate" -Dtests.seed=6D39D8439C254FF0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-VE -Dtests.timezone=Pacific/Honolulu -Druntime.java=17

All 3 failures were caused by "Suite timeout exceeded (>= 1200000 msec)."

From this I'll make a couple hypotheses:

1. There is a bug in the logic of ShardIndexingPressureSettingsIT that sometimes causes it to hang and fail with the overall test timeout. See previous issue where this same failure occurred.
2. While failures we see in the PR workflows that run ./gradlew check often manifest as a failure somewhere in :server:internalClusterTest, they are not the result of buggy logic within the tests themselves, but instead are the result of interference between gradle tasks running concurrently, or some other problem with the CI environment. (I make this claim because the ~2% failure rate observed in my experiment seems much lower than the failure rate we're observing in the PR checks)

I'm going to repeat my experiment but run the full check task instead of just :server:internalClusterTest. If hypothesis 2 is correct then I should see a higher failure rate than 3 out of 152 observed in this first experiment.

Dev environment:

• OS: Ubuntu 20.04
• Host type: c6i.8xlarge
• Branch: main at 309649ce8a

[saratvemulapalli]

Another flaky test:
Coming from: #1725

* What went wrong:
Execution failed for task ':qa:rolling-upgrade:v1.3.0#oldClusterTest'.
> `node{:qa:rolling-upgrade:v1.3.0-0}` failed to wait for ports files after 120000 MILLISECONDS

[dreamer-89]

Looking into it.

[dreamer-89]

A simple plan to begin with can involve below steps:

1. Analyze.
   Analyze last X failed Jenkins builds (X=20), identify failed tests and count frequency of failure. This will help in priortizing the right failure.

2. Reproduce.
   Failures identified above may need more deep dive for root causes; and also the ability to reproduce those failures locally. The expectation from this step is to have dev setup where failures can be replicated. Begin with targeted test (fast); if it does not help, run entire tests suite (slow). Failures may not always happen so need to repeat the tests multiple times as done by @andrross above. Replication may need setup similar to as used in Jenkins (worst case; have Jenkins setup). Add required logs wherever necessary to deep dive into the issue. Replication may discover new bugs/issues in tests, these failures should be properly documented and fixed as well in order to increase the overall tests stability.

3. Fix. Fixing tests depends on type of failure and can broadlly be classified in below categories. The step may run in sequence after step 2 or in parallel depending upon failure identified in step 1.
   a. True transient failures.
   Failures which are happen randomly and are out of our control. For e.g. nodes connection time out happening due to bad node, networking issue etc. The only fix in this case it to either increase corresponding parameters (timeout) or skip the test until a proper fix is identified.
   b. Setup related.
   There may be class of failures related to mis-configurations (bcwd compatibility tests etc) and easiest one to identify. These tests may need minor configuration changes.
   b. Bug fix.
   The remaining class of failures are corner cases which are more tricky root cause and may need specific area of expertise. Based on area of failure, required engineer needs to be involved to debug the issue further.

[andrross]

1. Analyze last X failed Jenkins builds (X=20)

I think it is a good idea to collect this data. It might be a bit hard to separate out the failures that were caused by the change in the PR that triggered the build. Setting up a test machine to run checks continually should be able to get similar data, and will have the benefit of running against a static code base.

2. Reproduce

We've probably seen enough of these to know they aren't reproducable when re-run in isolation. We have open issues with quite a few errors and none of them can be reproduced even when re-running the individual test many many times. I think running the entire test suite is the way to go, but we probably don't need to worry about the Jenkins stuff and can just trigger the ./gradlew check command directly.

[saratvemulapalli]

Another one, coming from: #1766

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.discovery.StableMasterDisruptionIT.testStaleMasterNotHijackingMajority" -Dtests.seed=28AD28E1A3FF50C7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-PH -Dtests.timezone=Etc/GMT+8 -Druntime.java=15

org.opensearch.discovery.StableMasterDisruptionIT > testStaleMasterNotHijackingMajority FAILED
    java.lang.AssertionError: node_t1: [Tuple [v1=node_t2, v2=null]]
        at __randomizedtesting.SeedInfo.seed([28AD28E1A3FF50C7:77AB65EE82248FCB]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.opensearch.discovery.StableMasterDisruptionIT.lambda$testStaleMasterNotHijackingMajority$5(StableMasterDisruptionIT.java:253)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1048)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1021)
        at org.opensearch.discovery.StableMasterDisruptionIT.testStaleMasterNotHijackingMajority(StableMasterDisruptionIT.java:250)

[andrross]

I ran another experiment over the weekend, the theory being that maybe :qa:mixed-cluster:v1.2.2#mixedClusterTest was interfering with :server:internalClusterTest:

for i in $(seq 0 1000) ; do echo "Iteration: $i" && ./gradlew clean > /dev/null 2>&1 && ./gradlew :server:internalClusterTest :qa:mixed-cluster:v1.2.2#mixedClusterTest >> ../build-failure-tests/test-output-2021-12-17_2.txt 2>&1 ; done

but the results were 7 failures out of 330, which is in line with the ~2% failure rate of the integ tests in isolation. The failures were:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.ClusterHealthIT.testHealthOnMasterFailover" -Dtests.seed=60436199814D8A58 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr-CS -Dtests.timezone=Etc/GMT+5 -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.ClusterHealthIT.testHealthOnMasterFailover" -Dtests.seed=8EC37C710AA42BCE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=no-NO -Dtests.timezone=EET -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.ClusterHealthIT.testHealthOnMasterFailover" -Dtests.seed=B4175006736B7460 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Africa/Casablanca -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites" -Dtests.seed=6AF32DFBEB864CEE -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hant-TW -Dtests.timezone=PRC -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites" -Dtests.seed=D921821394B6DBAA -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-GB -Dtests.timezone=America/Nipigon -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting" -Dtests.seed=FA529FAA49915455 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-SY -Dtests.timezone=AET -Druntime.java=17
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting" -Dtests.seed=FC550CFC70BBB318 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hans-CN -Dtests.timezone=America/Knox_IN -Druntime.java=17

There are likely bugs within ClusterHealthIT, ShardIndexingPressureIT, and ShardIndexingPressureSettingsIT that cause rare failures. But it remains a mystery what is causing ./gradlew check to fail at a much higher rate in the CI workflow than in these experiments.

[dblock]

#1725

I opened #1793 for this one specifically.