Flakey Tests

[stefanmb]

Please add new flakes to this spreadsheet

---

Original issue

Based on manual analysis of failed nightly builds:

• test_bad_container_on_stateful_sets_fails_with_rolling_update
  • This test seems 100% broken on k8s 1.9
  • It has also failed on 1.7
• test_crashing_container_on_deployment_fails_quickly
  • 1.8 fails sporadically
  • 1.6 fails sporadically
• test_deploy_result_logging_for_mixed_result_deploy
  • 1.8 fails sporadically
  • 1.6 fails sporadically
• test_deployment_with_progress_times_out_for_short_duration
  • 1.8 fails sporadically
• test_output_when_switching_labels_incorrectly
  • 1.6 fails sporadically

[KnVerey]

• test_bad_init_container_on_deployment_fails_quickly
  • 1.8 fails sporadically
• test_output_of_failed_unmanaged_pod
  • 1.8 failed once (missing "The following containers encountered errors:")
• test_bad_container_on_daemon_sets_fails
  • all versions fail sporadically 1.6, 1.7, 1.8
• test_restart_failure
  • 1.8 failed a couple times

[KnVerey]

Test	1.6	1.7	1.8	1.9	Notes	Fixed
test_bad_container_on_ stateful_sets_fails_ with_rolling_update		x		x	Always fails on 1.9. Trivial fix.
test_crashing_container_on_ deployment_fails_quickly	x		x		this passed 100 times in a row for me locally against 1.8, taking between 7s and 17s. On CI it fails regularly with a PDS timeout. When it passes, it does so in ~8 seconds, whereas it fails after ~22s.
test_deploy_result_logging_for _mixed_result_deploy	x		x		It's usually the init crash portion timing out instead of failing fast. Rarely, the container that is supposed to be running isn't.
test_deployment_with_progress _times_out_for_short_duration			x		This isn't a problem with the test, but the fixture, which needs its GITHUB_REV to start with a letter
test_output_when_switching _labels_incorrectly	x
test_bad_init_container_on _deployment_fails_quickly	x		x
test_output_of_failed _unmanaged_pod			x
test_bad_container_on_ daemon_sets_fails	x	x	x
test_restart_failure			x

[KnVerey]

Since I just put a bunch of work into addressing this, I've made a spreadsheet to help us track new flakes after my PR merges.