Add multiple endpoint listener for fluss server. #531

loserwang1024 · 2025-03-05T06:50:27Z

Purpose

Linked issue: close #421 , add multiple endpoint listener for fluss server.

Brief change log

Deprecated config:

coordinator.host
coordinator.port
tablet-server.host
tablet-server.port

Added config

bind.listeners: required, The network address and port the server binds to for accepting connections.
advertised.listeners: required, The externally advertised address and port for client connections.
internal.listener.name: required, and must be included in coordinator.listener and tablet-server.listeners. Otherwise, internal servers can not communicate.

Tests

API and Format

Documentation

loserwang1024 · 2025-03-05T06:51:15Z

@wuchong @leonardBang @swuferhong @luoyuxia , CC

loserwang1024 · 2025-03-13T15:34:28Z

@wuchong , rebase and refactor this pr, CC

wuchong

Please update server.yaml as well.
Please create an issue to update local_cluster.sh to adapt the config changes.
Please create an issue to update all the documentation.
Validate the configs, e.g, all listener names are not duplicated, ports are not duplicated. All the listener names in advertised.listeners and internal.listener.name can be found in bind.listeners? Take Kafka kafka.utils.CoreUtils.listenerListToEndPoints as an example, and add tests.

fluss-server/src/test/java/com/alibaba/fluss/server/testutils/FlussClusterExtension.java

fluss-common/src/test/java/com/alibaba/fluss/config/EndPointTest.java

fluss-server/src/test/java/com/alibaba/fluss/server/testutils/FlussClusterExtension.java

fluss-common/src/test/java/com/alibaba/fluss/config/EndPointTest.java

fluss-server/src/test/java/com/alibaba/fluss/server/coordinator/CoordinatorServerTest.java

fluss-server/src/test/java/com/alibaba/fluss/server/tablet/TabletServerTest.java

wuchong · 2025-03-14T14:04:09Z

fluss-server/src/main/java/com/alibaba/fluss/server/metadata/ClusterMetadataInfo.java

                                    pbCoordinatorServer.getNodeId(),
-                                    pbCoordinatorServer.getHost(),
-                                    pbCoordinatorServer.getPort(),
+                                    Endpoint.parseEndpoints(pbCoordinatorServer.getListeners()),


to support compatible with old version (host & prot).
and add UT for the compatibility test (similar to com.alibaba.fluss.server.coordinator.TableManagerITCase#testMetadata)

Add com.alibaba.fluss.server.coordinator.TableManagerITCase#testMetadataCompatibility

fluss-rpc/src/test/java/com/alibaba/fluss/rpc/netty/client/NettyClientTest.java

loserwang1024 · 2025-03-17T12:50:57Z

Add subtask to modify shell and docs
modify this pr based on cr.
@wuchong , CC

wuchong · 2025-03-20T09:59:23Z

fluss-common/src/main/java/com/alibaba/fluss/exception/EndpointNotAvailableException.java

+package com.alibaba.fluss.exception;
+
+/** Exception thrown when the endpoint is not available. */
+public class EndpointNotAvailableException extends ApiException {


This is only used in the server, so doesn't need to be an ApiException. Move it to com.alibaba.fluss.server.exception. Besides, ApiException strips the exception stack, which is very inconvenient to debug. Here it should extend RuntimeException.

wuchong · 2025-03-20T10:00:16Z

fluss-common/src/main/java/com/alibaba/fluss/cluster/Endpoint.java

+        return Objects.hash(host, port, listenerName);
+    }
+
+    public String connectionString() {


listenerString to keep align with toListenersString, fromListenersString.

wuchong · 2025-03-20T10:08:55Z

fluss-common/src/test/java/cluster/EndpointTest.java

+ * limitations under the License.
+ */
+
+package cluster;


wrong package

wuchong · 2025-03-20T13:41:15Z

fluss-server/src/test/java/com/alibaba/fluss/server/testutils/FlussClusterExtension.java

@@ -621,8 +670,24 @@ public Builder setClusterConf(Configuration clusterConf) {
            return this;
        }

+        /** Sets the listeners of tablet servers. */
+        public Builder setTabletServerListeners(String tabletServerListeners) {


Good refactor!

wuchong · 2025-03-20T13:55:11Z

fluss-rpc/src/main/proto/FlussApi.proto

-  required int32 port = 3;
+  // host and port is used for client.
+  optional string host = 2;
+  optional int32 port = 3;


we can't change required to optional as it breaks compatibility, new version server sent events to the old version server, the events will be failed to deserialize on the old version server.

wuchong · 2025-03-20T14:15:44Z

I pushed a commit to fix the remaining minor comments.

wuchong · 2025-03-21T11:27:10Z

@loserwang1024 I'm going to merge it. If there are any problems, we can fix in the following PRs.

loserwang1024 requested review from leonardBang, wuchong, luoyuxia and swuferhong March 5, 2025 06:50

loserwang1024 mentioned this pull request Mar 11, 2025

Introduce authenticate in Fluss. #577

Open

loserwang1024 force-pushed the add-multi-endpoint-tmp branch 5 times, most recently from 3390a81 to 3321cac Compare March 13, 2025 14:26

Fluss server support multiple endpoint.

a1662d1

loserwang1024 force-pushed the add-multi-endpoint-tmp branch from 3321cac to a1662d1 Compare March 13, 2025 14:59

wuchong requested changes Mar 14, 2025

View reviewed changes

wuchong mentioned this pull request Mar 15, 2025

Start kafka netty server in TabletServer #610

Merged

loserwang1024 force-pushed the add-multi-endpoint-tmp branch 2 times, most recently from 349a666 to 6a37208 Compare March 17, 2025 09:59

modified based on cr

6b61950

loserwang1024 force-pushed the add-multi-endpoint-tmp branch from 6a37208 to 6b61950 Compare March 17, 2025 11:42

loserwang1024 requested a review from wuchong March 17, 2025 12:51

improve codes from Jark

d97be8c

wuchong reviewed Mar 20, 2025

View reviewed changes

wuchong merged commit 157d252 into alibaba:main Mar 21, 2025
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multiple endpoint listener for fluss server. #531

Add multiple endpoint listener for fluss server. #531

loserwang1024 commented Mar 5, 2025 •

edited

Loading

loserwang1024 commented Mar 5, 2025

loserwang1024 commented Mar 13, 2025

wuchong left a comment

wuchong Mar 14, 2025

loserwang1024 Mar 17, 2025

loserwang1024 commented Mar 17, 2025

wuchong Mar 20, 2025

wuchong Mar 20, 2025

wuchong Mar 20, 2025

wuchong Mar 20, 2025

wuchong Mar 20, 2025

wuchong commented Mar 20, 2025

wuchong commented Mar 21, 2025

Add multiple endpoint listener for fluss server. #531

Add multiple endpoint listener for fluss server. #531

Conversation

loserwang1024 commented Mar 5, 2025 • edited Loading

Purpose

Brief change log

Tests

API and Format

Documentation

loserwang1024 commented Mar 5, 2025

loserwang1024 commented Mar 13, 2025

wuchong left a comment

Choose a reason for hiding this comment

wuchong Mar 14, 2025

Choose a reason for hiding this comment

loserwang1024 Mar 17, 2025

Choose a reason for hiding this comment

loserwang1024 commented Mar 17, 2025

wuchong Mar 20, 2025

Choose a reason for hiding this comment

wuchong Mar 20, 2025

Choose a reason for hiding this comment

wuchong Mar 20, 2025

Choose a reason for hiding this comment

wuchong Mar 20, 2025

Choose a reason for hiding this comment

wuchong Mar 20, 2025

Choose a reason for hiding this comment

wuchong commented Mar 20, 2025

wuchong commented Mar 21, 2025

loserwang1024 commented Mar 5, 2025 •

edited

Loading