System tests flakes #863
Description
Describe the bug
Cluster Operator system tests have had many "flaky failures" in CI pipeline. We need to understand if it's a matter of improving the tests or they are indeed regression.
Example failures from CI
PVC expansion failures the most popular flakes:
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8.8/builds/55
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8.8/builds/39
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-rabbitmq-master-otp-23/builds/57
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-rabbitmq-master-otp-23/builds/48
Failure when testing MQTT, STOMP, and stream protocol is another common one:
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8-latest/builds/61
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8-latest/builds/50
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8-latest/builds/46
Another failures:
https://hush-house.pivotal.io/teams/rabbitmq-for-kubernetes/pipelines/operator/jobs/system-tests-3.8-latest/builds/37
Additional context
Add any other context about the problem here.