[CI] IndexShard.assertPrimaryMode failed in test run

[spinscale]

See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/376/console

Was unable to reproduce this.

Even though some security tests are failing, the repeating error message in here looks like an allocation to me (please change owners if I am wrong here) - also I assigned distributed/distributed here, not sure if Allocation or Engine would be the right label.

  2> WARNING: Uncaught exception in thread: Thread[elasticsearch[node_s0][write][T#2],5,TGRP-AuditTrailTests]
  2> java.lang.AssertionError: shard [.security_audit_log-2018.12.18][0], node[kvUxC0s8Q0-b8SmK__CjPg], [P], recovery_source[
new shard recovery], s[INITIALIZING], a[id=vIJeauiPSaihV_k7GhsKSg], unassigned_info[[reason=INDEX_CREATED], at[2018-12-18T15:
06:33.173Z], delayed=false, allocation_status[no_attempt]] is not a primary shard in primary mode
  2>    at __randomizedtesting.SeedInfo.seed([B72A53FE1BA8D97E]:0)
  2>    at org.elasticsearch.index.shard.IndexShard.assertPrimaryMode(IndexShard.java:1542)
  2>    at org.elasticsearch.index.shard.IndexShard.getReplicationGroup(IndexShard.java:1963)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.getReplicationGroup(
TransportReplicationAction.java:1065)
  2>    at org.elasticsearch.action.support.replication.ReplicationOperation.checkActiveShardCount(ReplicationOperation.java:
221)
  2>    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:91)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:412)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:358)
  2>    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
  2>    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:269)
  2>    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:236)
  2>    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2325)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:969)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:357)
  2>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:312)
  2>    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:304)
  2>    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)
  2>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>    at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:140)
  2>    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:300)
  2>    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
  2>    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:135)
  2>    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:177)
  2>    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:171)
  2>    at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:159)
  2>    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:137)
  2>    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:180)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:213)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:178)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:137)
  2>    at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:107)
  2>    at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:119)
  2>    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307)
  2>    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
  2>    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:688)
  2>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
  2>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>    at java.lang.Thread.run(Thread.java:748)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/376/consoleText

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

I think this is one of the many very slow test runs we've seen this week. There's a bunch of failures in this run as well as the build OOMing with error code 137 (seems this is a CI infrastructure issue since we're getting killed by the kernel):

* What went wrong:
Execution failed for task ':x-pack:plugin:security:test'.
> At least one slave process threw an exception, first: Forked process returned with error code: 137. Very likely a JVM crash.  See process stderr at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+periodic/x-pack/plugin/security/build/testrun/test/temp/junit4-J3-20181218_150323_79010427487910268165883.syserr

=> I'd just close this.

[original-brownbear]

@atorok I'm seeing the test JVMs die with 137 here and in one other issue. Do you know if this is something known or should I raise an issue about that with infra?

[alpar-t]

This happened a non ephemeral CI worker, so it is possible something is leaking memory on it. I don't know of existing investigations or issues.

[original-brownbear]

The failure hasn't happened again since this report and there was obviously something fishy going on in this run memory wise (see above #36783 (comment)) => I'm closing this one.