[RFC] When to retire the serialization backward compat code for OpenSearch 2.11/2.12/2.13

[nibix]

In OpenSearch 2.11, a custom serialization mechanism for User objects and other serialized info was introduced. Due to unforeseen issues, it had to be rolled back in OpenSearch 2.14. See #4264 for the change that did the rollback. See #4670 for an example of the issues caused by 2.11/2.12/2.13 .

There is still compatibility code which allows mixed clusters with OpenSearch versions 2.11/2.12/2.13 to co-exist with nodes with newer nodes.

My question is: When can we remove this compatibility mode?

In other words: Which OpenSearch version shall be no longer mixable with 2.11/2.12/2.13? Is it OpenSearch 3?

[nibix]

Background: This is related to the "Immutable user object" effort #5168 and intended to make the logic and the code more simple.

[cwperks]

My question is: When can we remove this compatibility mode?

It can safely be removed in 3.0.0. The upgrade path for OpenSearch requires going to the last minor version of the previous major before going to the next major version for rolling upgrades.

Its not possible to mix 2.11/2.12/2.13 nodes with 3.0.0 nodes, my understanding is that the 3.0.0 would refuse to join the cluster.

To upgrade from 2.11 -> 3.0 with rolling upgrades would require 2.11 -> 2.19 -> 3.0.0

[nibix]

Confirmed by executing

        System.out.println("minimumCompatibilityVersion: " + Version.V_3_0_0.minimumCompatibilityVersion());

minimumCompatibilityVersion: 2.19.0

[shikharj05]

In other words: Which OpenSearch version shall be no longer mixable with 2.11/2.12/2.13? Is it OpenSearch 3?

I think we have the answer above for upgrades. Do we know about Cross-cluster search? @nibix @cwperks

[nibix]

@shikharj05

Do we know about Cross-cluster search? @nibix @cwperks

Very good question! I don't have an answer right away. cc @kumargu

[shikharj05]

currently, cross-cluster replication only checks that leader version should be lower than follower version.

https://github.com/opensearch-project/cross-cluster-replication/blob/main/src/main/kotlin/org/opensearch/replication/util/ValidationUtil.kt#L140-L153

[nibix]

@shikharj05 I do not know so much about CCR. Does it also communicate via the transport protocol?

Just now, we have tested CCS between clusters on OpenSearch 2.18.x and OpenSearch 3.0.0.

This actually does not work; OpenSearch applies the same restrictions as for nodes within a single cluster. The error:

{
  "error": {
    "root_cause": [
      {
        "type": "connect_transport_exception",
        "reason": "[][172.18.0.7:9300] general node connection failure"
      }
    ],
    "type": "connect_transport_exception",
    "reason": "[][172.18.0.7:9300] general node connection failure",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "Received message from unsupported version: [2.18.0] minimal compatible version is: [2.19.0]"
    }
  },
  "status": 500
}

Thus, only OpenSearch 2.19.x can interoperate with OpenSearch 3.0.

[shikharj05]

@shikharj05 I do not know so much about CCR. Does it also communicate via the transport protocol?

Yes, it uses transport layer. If security is enabled, you need to setup nodes_dn, transport certificate trust, etc - https://docs.opensearch.org/docs/latest/tuning-your-cluster/replication-plugin/getting-started/#permissions

Thanks @nibix for the quick test! qq - was this search executed with 2.18 as source or destination? If 2.18 was source, what if 3.0 is source cluster? From what I know, the source version must be same or higher than destination version.

[AntonEliatra]

Tested with source of 2.18 and remote of 3.0, see response below:

{
  "error": {
    "root_cause": [
      {
        "type": "node_disconnected_exception",
        "reason": "[][172.19.0.3:9300][internal:transport/handshake] disconnected"
      }
    ],
    "type": "node_disconnected_exception",
    "reason": "[][172.19.0.3:9300][internal:transport/handshake] disconnected"
  },
  "status": 500
}

Further logs as follows: (opensearch-cluster2-node1 is 3.0)

opensearch-cluster2-node1       | [2025-04-17T06:42:38,172][ERROR][o.o.t.n.s.SecureNetty4Transport] [opensearch-cluster2-node1] Exception during establishing a SSL connection: java.lang.IllegalStateException: Received message from unsupported version: [2.18.0] minimal compatible version is: [2.19.0]
opensearch-cluster2-node1       | java.lang.IllegalStateException: Received message from unsupported version: [2.18.0] minimal compatible version is: [2.19.0]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.ensureVersionCompatibility(InboundDecoder.java:228) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.readHeader(InboundDecoder.java:196) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:95) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.decode(InboundDecoder.java:73) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:61) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[transport-netty4-client-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-cluster2-node1       | [2025-04-17T06:42:38,174][WARN ][o.o.t.TcpTransport       ] [opensearch-cluster2-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.19.0.3:9300, remoteAddress=/172.19.0.2:56988}], closing connection
opensearch-cluster2-node1       | java.lang.IllegalStateException: Received message from unsupported version: [2.18.0] minimal compatible version is: [2.19.0]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.ensureVersionCompatibility(InboundDecoder.java:228) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.readHeader(InboundDecoder.java:196) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.internalDecode(InboundDecoder.java:95) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundDecoder.decode(InboundDecoder.java:73) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundBytesHandler.doHandleBytes(InboundBytesHandler.java:61) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:124) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:113) ~[opensearch-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[transport-netty4-client-3.0.0-alpha1.jar:3.0.0-alpha1]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.118.Final.jar:4.1.118.Final]
opensearch-cluster2-node1       | 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
opensearch-cluster1-node1       | [2025-04-17T06:42:38,176][WARN ][o.o.t.SniffConnectionStrategy] [opensearch-cluster1-node1] fetching nodes from external cluster [opensearch-ccs-cluster1] failed
opensearch-cluster1-node1       | org.opensearch.transport.NodeDisconnectedException: [][172.19.0.3:9300][internal:transport/handshake] disconnected
opensearch-cluster1-node1       | [2025-04-17T06:42:38,178][WARN ][r.suppressed             ] [opensearch-cluster1-node1] path: /opensearch-ccs-cluster1:meetup/_search, params: {pretty=true, index=opensearch-ccs-cluster1:meetup}
opensearch-cluster1-node1       | org.opensearch.transport.RemoteTransportException: [error while communicating with remote cluster [opensearch-ccs-cluster1]]
opensearch-cluster1-node1       | Caused by: org.opensearch.transport.NodeDisconnectedException: [][172.19.0.3:9300][internal:transport/handshake] disconnected
opensearch-cluster1-dashboards  | {"type":"log","@timestamp":"2025-04-17T06:42:38Z","tags":["error","opensearch","data"],"pid":1,"message":"[node_disconnected_exception]: [][172.19.0.3:9300][internal:transport/handshake] disconnected"}

[AntonEliatra]

In the first case that @nibix reported, 2.18 was the remote and 3.0 coordinating.