[CI] IndexShard.assertPrimaryMode error in :x-pack:qa:rolling-upgrade-multi-cluster

[davidkyle]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/334/console

Execution failed for task ':x-pack:qa:rolling-upgrade-multi-cluster:v6.7.0#leader#oneThirdUpgradedTestCluster#wait'.

The node was taken down by the assertion

Caused by: java.lang.AssertionError: shard [logs-20200101][0], node[CvprUfVxSxazBWFEnwpJ0g], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=2q9Hz_lnTtqVS07KFe0Wjw], unassigned_info[[reason=NODE_LEFT], at[2019-04-30T11:58:34.033Z], delayed=true, details[node_left [CvprUfVxSxazBWFEnwpJ0g]], allocation_status[fetching_shard_data]] is not a primary shard in primary mode
|    	at org.elasticsearch.index.shard.IndexShard.assertPrimaryMode(IndexShard.java:1632) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShard.renewRetentionLease(IndexShard.java:2030) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$Renew$TransportAction.doRetentionLeaseAction(RetentionLeaseActions.java:248) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$Renew$TransportAction.doRetentionLeaseAction(RetentionLeaseActions.java:222) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction$1.onResponse(RetentionLeaseActions.java:115) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction$1.onResponse(RetentionLeaseActions.java:110) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction.asyncShardOperation(RetentionLeaseActions.java:109) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction.asyncShardOperation(RetentionLeaseActions.java:65) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:296) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:289) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1046) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:932) ~[elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) [elasticsearch-6.7.2-SNAPSHOT.jar:6.7.2-SNAPSHOT]
|    	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
|    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
|    	... 22 more

Possibly related to #36783 although the stack trace is different

[elasticmachine]

Pinging @elastic/es-distributed

[benwtrent]

This happened again upgrading from 6.8 to 7.2.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+artifactory/640/console

Failed node was the newly added upgraded node (running 7.2).

[2019-05-08T11:24:20,786][WARN ][o.e.t.TcpTransport       ] [upgraded-node-leader-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:50204, remoteAddress=/127.0.0.1:41128}], closing connection
java.lang.Exception: java.lang.AssertionError: shard [logs-20200102][0], node[CHjjB1z5SSGaEHm2ydaNFA], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=h6QgAFYsQ8qHKejNqFQ4Gg], unassigned_info[[reason=NODE_LEFT], at[2019-05-08T11:23:55.160Z], delayed=true, details[node_left [CHjjB1z5SSGaEHm2ydaNFA]], allocation_status[fetching_shard_data]] is not a primary shard in primary mode
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:75) [transport-netty4-client-7.2.0-SNAPSHOT.jar:7.2.0-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:836) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

Looking at the logs from the non-upgraded leader nodes, they seem to simply see the upgraded node.

[2019-05-08T11:24:21,139][WARN ][o.e.c.NodeConnectionsService] [node-2] failed to connect to node {upgraded-node-leader-0}{CHjjB1z5SSGaEHm2ydaNFA}{Plx95G1GQyev0km4RnBthw}{127.0.0.1}{127.0.0.1:50204}{testattr=test, upgraded=true, xpack.installed=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [upgraded-node-leader-0][127.0.0.1:50204] general node connection failure
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener$1.onFailure(TcpTransport.java:1294) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:155) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:67) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.lambda$wrap$0(ActionListener.java:83) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:97) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:39) ~[elasticsearch-core-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_212]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_212]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_212]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_212]
	at org.elasticsearch.common.concurrent.CompletableContext.complete(CompletableContext.java:61) ~[elasticsearch-core-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$0(Netty4TcpChannel.java:51) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1152) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:768) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:744) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:615) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1376) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.close(LoggingHandler.java:217) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[?:?]
	at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:1003) ~[?:?]
	at io.netty.channel.AbstractChannel.close(AbstractChannel.java:242) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.close(Netty4TcpChannel.java:80) ~[?:?]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85) ~[elasticsearch-core-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.common.network.CloseableChannel.closeChannels(CloseableChannel.java:89) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.common.network.CloseableChannel.closeChannel(CloseableChannel.java:78) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.common.network.CloseableChannel.closeChannel(CloseableChannel.java:68) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.onException(TcpTransport.java:604) ~[elasticsearch-6.8.0-SNAPSHOT.jar:6.8.0-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:68) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) ~[?:?]
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.exceptionCaught(LoggingHandler.java:175) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1401) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:953) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.elasticsearch.transport.TransportException: handshake failed because connection reset
	... 64 more
[2019-05-08T11:24:21,148][INFO ][o.e.c.s.ClusterApplierService] [node-2] removed {{upgraded-node-leader-0}{CHjjB1z5SSGaEHm2ydaNFA}{Plx95G1GQyev0km4RnBthw}{127.0.0.1}{127.0.0.1:50204}{testattr=test, upgraded=true, xpack.installed=true},}, reason: apply cluster state (from master [master {node-1}{KegPAxwbSbS-zv5ADynsEw}{H5y9wNdcRu-9_ymvj7NJKA}{127.0.0.1}{127.0.0.1:39838}{testattr=test, xpack.installed=true} committed version [42]])

[davidkyle]

Another https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/367/console

Execution failed for task ':x-pack:qa:rolling-upgrade-multi-cluster:v6.7.0#leader#oneThirdUpgradedTestCluster#stop'

|    Caused by: java.lang.AssertionError: shard [logs-20200102][0], node[i0PN0hEEQnGkneIA7_SIDA], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=shaLFBj8TCSwRRwXeYelNg], unassigned_info[[reason=NODE_LEFT], at[2019-05-09T12:45:43.299Z], delayed=true, details[node_left [i0PN0hEEQnGkneIA7_SIDA]], allocation_status[fetching_shard_data]] is not a primary shard in primary mode
|    	at org.elasticsearch.index.shard.IndexShard.assertPrimaryMode(IndexShard.java:1632) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShard.renewRetentionLease(IndexShard.java:2030) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$Renew$TransportAction.doRetentionLeaseAction(RetentionLeaseActions.java:248) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$Renew$TransportAction.doRetentionLeaseAction(RetentionLeaseActions.java:222) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction$1.onResponse(RetentionLeaseActions.java:115) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction$1.onResponse(RetentionLeaseActions.java:110) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction.asyncShardOperation(RetentionLeaseActions.java:109) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.index.seqno.RetentionLeaseActions$TransportRetentionLeaseAction.asyncShardOperation(RetentionLeaseActions.java:65) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:296) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:289) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1046) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:932) ~[elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) [elasticsearch-6.7.3-SNAPSHOT.jar:6.7.3-SNAPSHOT]
|    	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
|    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]