CI: SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions fails

[alpar-t]

Example build failure

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/162/

There are other issues for this tetst: #30777 and #29893

Reproduction line

does not reproduce locally

./gradlew :x-pack:qa:smoke-test-watcher-with-security:integTestRunner \
  -Dtests.seed=98BA944A70941ADF \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT \
  -Dtests.method="testSearchTransformInsufficientPermissions" \
  -Dtests.security.manager=true \
  -Dtests.locale=de-AT \
  -Dtests.timezone=Etc/Universal \
  -Dcompiler.java=11 \
  -Druntime.java=8

Example relevant log:

test log

17:22:27 > Throwable #1: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43210], URI [/.watcher-history-*/_search], status line [HTTP/1.1 503 Service Unavailable] 17:22:27 > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503} 17:22:27 > at __randomizedtesting.SeedInfo.seed([98BA944A70941ADF:5CF90B4519F1D32]:0) 17:22:27 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+intake/x-pack/qa/smoke-test-watcher-with-security/build/testrun/integTestRunner/J0/temp/org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT_98BA944A70941ADF-001 17:22:27 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1435, maxMBSortInHeap=7.554326604207141, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@39e7681e), locale=de-AT, timezone=Etc/Universal 17:22:27 > at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:687) 17:22:27 > at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.lambda$getWatchHistoryEntry$3(SmokeTestWatcherWithSecurityIT.java:325) 17:22:27 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836) 17:22:27 2> NOTE: Linux 3.10.0-862.14.4.el7.x86_64 amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=435688080,total=514850816 17:22:27 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822) 17:22:27 2> NOTE: All tests run in this JVM: [SmokeTestWatcherWithSecurityIT] 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:306) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.getWatchHistoryEntry(SmokeTestWatcherWithSecurityIT.java:301) 17:22:27 > at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformInsufficientPermissions(SmokeTestWatcherWithSecurityIT.java:237) 17:22:27 > at java.lang.Thread.run(Thread.java:748) 17:22:27 > Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:43210], URI [/.watcher-history-*/_search], status line [HTTP/1.1 503 Service Unavailable] 17:22:27 > {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503} 17:22:27 > at org.elasticsearch.client.RestClient$1.completed(RestClient.java:307) 17:22:27 > at org.elasticsearch.client.RestClient$1.completed(RestClient.java:292) 17:22:27 > at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) 17:22:27 > at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) 17:22:27 > at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) 17:22:27 > at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) 17:22:27 > at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) 17:22:27 > at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) 17:22:27 > at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) 17:22:27 > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) 17:22:27 > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) 17:22:27 > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) 17:22:27 > ... 1 more 17:22:27 Completed [1/2] in 18.03s, 6 tests, 1 error <<< FAILURES!

server

17:22:45 [2018-11-07T15:22:23,953][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [mWqFTVwuOPGWaKGubYkz/index] 17:22:45 org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager] 17:22:45 at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:580) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:555) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:299) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:386) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:499) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:238) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:164) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:88) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:389) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:378) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:397) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.lambda$execute$0(ExecutableIndexAction.java:100) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:122) ~[x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.execute(ExecutableIndexAction.java:99) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) [x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:455) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:395) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:544) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 18 LINES ... 17:22:45 [2018-11-07T15:22:26,492][WARN ][r.suppressed ] [node-0] path: /.watcher-history-*/_search, params: {index=.watcher-history-*} 17:22:45 org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:106) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:277) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:440) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1127) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1244) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1218) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onFailure(HandledTransportAction.java:102) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:348) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:342) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:336) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1075) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:27,434][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_9a9dc0c0-31f0-4ba8-97cd-fc2b896c6e1a-2018-11-07T15:22:27.430Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 65 LINES ... 17:22:45 [2018-11-07T15:22:30,649][ERROR][o.e.x.w.t.s.ExecutableSearchTransform] [node-0] failed to execute [search] transform for [_inlined__0dd49d6b-5069-41e4-bc95-7c9f3dbd5154-2018-11-07T15:22:30.646Z] 17:22:45 org.elasticsearch.common.ParsingException: no [query] registered for [does_not_exist] 17:22:45 at org.elasticsearch.index.query.AbstractQueryBuilder.parseInnerQueryBuilder(AbstractQueryBuilder.java:326) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:1044) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:989) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.support.search.WatcherSearchTemplateService.toSearchRequest(WatcherSearchTemplateService.java:70) ~[x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transform.search.ExecutableSearchTransform.execute(ExecutableSearchTransform.java:51) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transform.search.ExecutableSearchTransform.execute(ExecutableSearchTransform.java:27) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:444) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:149) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 17:22:45 at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 ... SKIPPED 51 LINES ... 17:22:45 [2018-11-07T15:22:32,861][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_be0e7718-f474-43e4-b303-27c4809b48b5-2018-11-07T15:22:32.857Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 12 LINES ... 17:22:45 [2018-11-07T15:22:33,320][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_c791e726-58ee-4d44-8444-e26449d5a826-2018-11-07T15:22:33.133Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:33,399][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_12ce4688-7a30-4944-84ae-2061df2a7a49-2018-11-07T15:22:33.380Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:33,444][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_8d8bc3f3-94c0-45a9-a0b0-ba841ac17243-2018-11-07T15:22:33.419Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 15 LINES ... 17:22:45 [2018-11-07T15:22:33,999][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_ba8356d3-b059-4893-894f-d32f61a8b147-2018-11-07T15:22:33.927Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:34,389][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_7e085d94-cfba-4eae-adae-2fde7c41b79d-2018-11-07T15:22:34.364Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 21 LINES ... 17:22:45 [2018-11-07T15:22:35,171][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_9cf2df9c-5279-45cd-a6b6-cb5d2960ae56-2018-11-07T15:22:35.084Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:35,523][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_f1417293-e22c-4f2f-b9b4-0beae3ba2858-2018-11-07T15:22:35.521Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 21 LINES ... 17:22:45 [2018-11-07T15:22:36,291][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_44475776-4c29-4e35-9d32-bccfb3cb725d-2018-11-07T15:22:36.214Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 11 LINES ... 17:22:45 [2018-11-07T15:22:36,659][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_a1d347a5-7f2a-4af4-ba7f-af60a08e7c21-2018-11-07T15:22:36.656Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 17 LINES ... 17:22:45 [2018-11-07T15:22:37,251][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_5c6fb0f8-3856-472c-83bc-a2eef30b5eb2-2018-11-07T15:22:37.181Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 14 LINES ... 17:22:45 [2018-11-07T15:22:38,680][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_ee71191e-4d90-4349-931a-d5f927417296-2018-11-07T15:22:37.659Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 42 LINES ... 17:22:45 [2018-11-07T15:22:40,236][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_b9d98c2e-20ea-4363-9dbc-aff7cf00a985-2018-11-07T15:22:40.235Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:40,242][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_c87d0280-3af9-4551-bcd8-4897a313476c-2018-11-07T15:22:40.241Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:40,611][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_c33163fe-7ff5-426a-b602-41d70f92da42-2018-11-07T15:22:40.585Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 27 LINES ... 17:22:45 [2018-11-07T15:22:41,584][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_b4833751-f2c7-46cc-a1da-c99af56f0745-2018-11-07T15:22:41.416Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:41,648][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_b5b0be44-b074-4d11-a961-9a07165d5f69-2018-11-07T15:22:41.630Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:41,691][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_10d43a66-61cb-4f02-9573-d891d419831f-2018-11-07T15:22:41.668Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 18 LINES ... 17:22:45 [2018-11-07T15:22:42,327][ERROR][o.e.x.w.a.i.IndexActionFactory] [node-0] failed to execute action [my_watch/index] 17:22:45 org.elasticsearch.ElasticsearchSecurityException: action [indices:data/write/bulk[s]] is unauthorized for user [watcher_manager] 17:22:45 at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:30) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:580) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.denial(AuthorizationService.java:555) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:299) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:386) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:499) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.executeIngestAndBulk(TransportBulkAction.java:238) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:164) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:88) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:141) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:90) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:179) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:175) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:169) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:157) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:181) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) ~[?:?] 17:22:45 at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:211) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:222) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:176) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:136) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:106) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:158) ~[?:?] 17:22:45 at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:111) ~[?:?] 17:22:45 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:139) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:117) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:60) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:389) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:378) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:397) ~[elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.lambda$execute$0(ExecutableIndexAction.java:100) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.ClientHelper.executeWithHeaders(ClientHelper.java:122) ~[x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.actions.index.ExecutableIndexAction.execute(ExecutableIndexAction.java:99) ~[?:?] 17:22:45 at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) [x-pack-core-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:455) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:149) [x-pack-watcher-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] 17:22:45 at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 17:22:45 at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:627) [elasticsearch-7.0.0-alpha1-SNAPSHOT.jar:7.0.0-alpha1-SNAPSHOT] 17:22:45 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 17:22:45 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 17:22:45 at java.lang.Thread.run(Thread.java:834) [?:?] 17:22:45 [2018-11-07T15:22:42,330][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_3063ff1f-3667-41eb-b1c6-e6bf66ddee50-2018-11-07T15:22:42.326Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 8 LINES ... 17:22:45 [2018-11-07T15:22:42,583][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_2492079c-a958-46e5-b15e-d44347cabab7-2018-11-07T15:22:42.583Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 3 LINES ... 17:22:45 [2018-11-07T15:22:42,720][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_1ac97e9e-d648-41d7-b19c-34f6beb8262c-2018-11-07T15:22:42.611Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 [2018-11-07T15:22:42,766][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [test_watch_9264c270-5da1-4e3c-9d75-46132f5c192b-2018-11-07T15:22:42.748Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 10 LINES ... 17:22:45 [2018-11-07T15:22:43,095][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_0a38706d-af18-4cd9-8393-a04d29f30381-2018-11-07T15:22:43.026Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 ... SKIPPED 15 LINES ... 17:22:45 [2018-11-07T15:22:43,671][ERROR][o.e.x.w.Watcher ] [node-0] triggered watches could not be deleted [my_watch_36d41cd5-379d-4bb9-b7a3-4babe24cd2de-2018-11-07T15:22:43.646Z], failure [[.triggered_watches] IndexNotFoundException[no such index [.triggered_watches]]] 17:22:45 =========================================

The test also seems to fail with:

Expected: is a value equal to or greater than <1>
     but: <0> was less than <1>

Similar to #33291 /cc @jakelandis

Frequency

1-2 times / week, increased to 4 this week.

Going to mute the test.

[elasticmachine]

Pinging @elastic/es-core-features

[alpar-t]

It seems like #33291 fixed the original assertion but resulted in a new failure.

[jakelandis]

@atorok
It looks like the build failure linked here had not included the fix for this yet.

Latest commit in the build : e685cfe
Commit that should fix this: d7a4cef

That build was 15 commits too early.

git rev-list --ancestry-path e685cfe..d7a4cef | wc -l
      15

Can we close this and un-mute the test.

FWIW, I am not 100% positive that the fix is a complete, but it does eliminate a race condition which should reduce some (hopefully all) of the noise from this test.

[alpar-t]

I didn't mean to imply that your changes caused new issues, I was thinking they uncovered it by going past the assert and running into

method [POST], host [http://[::1]:43210], URI [/.watcher-history-*/_search], status line [HTTP/1.1 503 Service Unavailable] {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}

But the last build on master that failed like this was before your changes.
I'll unmute the test to get more runs with your changes and see if this test has two separate
problems or the same manifests in different ways.

[jaymode]

Build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/55/consoleFull seems to have similar failures