High level REST client test framework encourages using randomness for test coverage #39667
Description
We want to avoid using randomness for test coverage. Randomness is good in cases when the values are not expected to impact the behavior of the functionality under test (e.g., randomizing the number of shards, whether or not replicas are present, or the store type in most cases is not expected to have any impact on functionality). In these cases, using randomness ensures that we are in fact not relying on implementation details of components unrelated to the functionality under test. However we want to avoid using randomness to ensure branch coverage, as it leads to tests that only run a fraction of the time, and so we can end up with a test passing (e.g., on a PR), only to fail later in CI. Instead, we want distinct test cases.
The high level REST client test framework encourages using randomness to ensure test coverage of sync versus async method execution. That is, the test framework randomly tests sync as opposed to async methods, only half the time covering one and half the time the other. Some tests that are invoking multiple high level REST clients in a single test end up with a confusing mix of sync and async requests.
Instead, we would want to refactor these tests so that on every test run we are covering both sync and async methods.