[CI] HttpInputIntegrationTests.testHttpInputClusterStats failed in 6.7

[droberts195]

HttpInputIntegrationTests.testHttpInputClusterStats failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+release-tests/220/console

The exception was:

   > Throwable #1: java.lang.AssertionError: could not find executed watch record for watch _name
   > Expected: a value equal to or greater than <1L>
   >      but: <0L> was less than <1L>
   > 	at __randomizedtesting.SeedInfo.seed([5630ED2413A03214:BB1CB2535603B200]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumPerformedActionsCount$2(AbstractWatcherIntegrationTestCase.java:396)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:858)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:832)
   > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.assertWatchWithMinimumPerformedActionsCount(AbstractWatcherIntegrationTestCase.java:378)
   > 	at org.elasticsearch.xpack.watcher.input.http.HttpInputIntegrationTests.testHttpInputClusterStats(HttpInputIntegrationTests.java:97)
   > 	at java.lang.Thread.run(Thread.java:748)

The problem does not reproduce locally using:

./gradlew :x-pack:plugin:watcher:unitTest \
  -Dtests.seed=5630ED2413A03214 \
  -Dtests.class=org.elasticsearch.xpack.watcher.input.http.HttpInputIntegrationTests \
  -Dtests.method="testHttpInputClusterStats" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=de-DE \
  -Dtests.timezone=ACT \
  -Dcompiler.java=12 \
  -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-core-features

[martijnvg]

I've muted (^) this test on 6.7 branch, so that I can merge #40681. This test kept failing in pr builds.

[gwbrown]

This should be fixed in master by #40658, if anyone runs across this failure let me know. I'll wait for the backport before closing this.

[gwbrown]

#40658 has been merged to all currently maintained branches after a week of running in master with no failures in this test, so I'm closing this issue.