[CI] NodeTests failing with incompatible wire format error

[cbuescher]

Could not reproduce, haven't seen any recent failures so far either so keeping this unmuted for now.

Build scan:
https://gradle-enterprise.elastic.co/s/auabxhkpzbxn2/tests/:server:test/org.elasticsearch.node.NodeTests/testCloseOnInterruptibleTask

Reproduction line:
./gradlew ':server:test' --tests "org.elasticsearch.node.NodeTests.testCloseOnInterruptibleTask" -Dtests.seed=3E36C1C728F289EA -Dtests.locale=pt-PT -Dtests.timezone=America/Ensenada -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.node.NodeTests&tests.test=testCloseOnInterruptibleTask

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=851, name=Thread-64, state=RUNNABLE, group=TGRP-NodeTests]


  Caused by: java.lang.AssertionError: java.lang.IllegalArgumentException: remote node [{TS_B}{TS_B}{HlNEkIWGTwKCjy--a1FF3w}{127.0.0.1}{127.0.0.1:9302}] is build [unknown-modified] of version [8.1.0] but this node is build [unknown] of version [8.1.0] which has an incompatible wire format

    at __randomizedtesting.SeedInfo.seed([3E36C1C728F289EA]:0)
    at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:346)
    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:833)

    Caused by: java.lang.IllegalArgumentException: remote node [{TS_B}{TS_B}{HlNEkIWGTwKCjy--a1FF3w}{127.0.0.1}{127.0.0.1:9302}] is build [unknown-modified] of version [8.1.0] but this node is build [unknown] of version [8.1.0] which has an incompatible wire format

      at org.elasticsearch.transport.TransportService$HandshakeResponse.<init>(TransportService.java:582)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:58)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:25)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1330)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1317)
      at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:338)
      at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
      at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
      at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796)
      at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
      at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
      at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
      at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[davidkyle]

Same error in NodeTests::testCloseRaceWithTaskExecution

Caused by: java.lang.AssertionError: java.lang.IllegalArgumentException: remote node [{TS_A}{TS_A}{negx7ZSKRuGTC2fd7P-Jsw}{127.0.0.1}{127.0.0.1:9301}] is build [unknown-modified] of version [8.1.0] but this node is build [unknown] of version [8.1.0] which has an incompatible wire format	

https://gradle-enterprise.elastic.co/s/d3u4xxger35yy/tests/:server:test/org.elasticsearch.node.NodeTests/testCloseRaceWithTaskExecution?top-execution=1

[ywangd]

Happened again for a different test method NodeTests testCloseOnLeakedIndexReaderReference

Build scan: https://gradle-enterprise.elastic.co/s/r3armmzv2bdjo

[ywangd]

Yet again for a different test method: testAwaitCloseTimeoutsOnNonInterruptibleTask

Build scan: https://gradle-enterprise.elastic.co/s/vxsveb6ihdugu/tests/:server:test/org.elasticsearch.node.NodeTests/testAwaitCloseTimeoutsOnNonInterruptibleTask?top-execution=1

[gmarouli]

Same error: https://gradle-enterprise.elastic.co/s/tve6z63uawuak/tests/:server:test/org.elasticsearch.node.NodeTests/testCloseRaceWithTaskExecution?top-execution=1

[nik9000]

testCloseOnInterruptibleTask for me https://gradle-enterprise.elastic.co/s/i6gbsfpbq5rcy/tests/:server:test/org.elasticsearch.node.NodeTests/testCloseOnInterruptibleTask?top-execution=1

[tlrx]

Happened again https://gradle-enterprise.elastic.co/s/epj3hcd3dpfzm

The tests in NodeTests started to fail after we started to use Netty4 transport more generally in tests (#82088). Surprisingly the tests fail because they connect to nodes that have TS_A or TS_B as names and that are likely to be started by other tests that run concurrently. I don't know enough of the transport service here to investigate, but maybe @original-brownbear has an idea?

I'm going to mute the whole test suite here.