test: Plumb the flaky flag from envoy_cc_test to the native.cc_test

[yanavlasov]

This flag allows test to temporarily be marked flaky while test flakiness is being diagnosed and fixed, to reduce the likelihood of CI failing.

See https://docs.bazel.build/versions/2.0.0/be/common-definitions.html#common-attributes-tests for more info about the flaky flag.

Risk Level: Low
Testing: Unit Tests
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Yan Avlasov yavlasov@google.com

[zuercher]

Nice.

[zuercher]

Assigned @lizan in case there's some reason we haven't done this earlier.

[mattklein123]

This seems like a good thing to have, though on the flip side it would probably mean we would never fix our flakes so I'm on the fence?

[lizan]

Do we want to use --runs_per_test_detects_flakes=true and --flaky_test_attempts as well? So it can give FLAKY status on CI.

[yanavlasov]

@mattklein123 It maybe a better alternative to disabling the test, which is what happens when the reason for flakiness is not obvious or time consuming to fix. Disabled test could get stuck in limbo juts as the flaky attribute. The only difference is that if someone breaks the test outright it would fail even if marked flaky, but if it is disabled, nobody would know.

@lizan I think --runs_per_test_detects_flakes=true is only useful with the --runs_per_test=nnn flag which we do not use in CI.
I would argue against --flaky_test_attempts because it hides newly flaky tests by making CI pass. It could be an option to control CI cost if a lot of CI failures are due to newly flaky tests, so if there is a way to analyze this, this could be a good case for it.

[lizan]

Nm, my idea was using both of --runs_per_test_detects_flakes=true and --flaky_test_attempts so we can distinct FAILED and FLAKY, but seems FLAKY will make CI pass.

[lizan]

Let's merge this first, we can decide whether we want to use this later when we mark tests flaky. I'm trying to use this to make coverage less flaky.