KAFKA-18028: the effective kraft version of --no-initial-controllers should be 1 rather than 0 #17836

FrankYang0529 · 2024-11-17T03:06:09Z

The hasDynamicQuorum only considers the initialControllers, which is configured when either --initial-controllers or --standalone is specified. It will result in the following unexpected behaviors:

Using the no-initial-controllers configuration get kraft.version=0
This leads to a command error when executing --no-initial-controllers --feature kraft.version=1

> ./bin/kafka-storage.sh format -t "zuV6tFqAQhSGfFe97O-9cw" --config ./config/kraft/controller3.properties --no-initial-controllers --feature kraft.version=1
Cannot set kraft.version to 1 unless KIP-853 configuration is present. Try removing the --feature flag for kraft.version.

To fix this, we should also allow --no-initial-controllers with --feature kraft.version=1.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ahuang98

thanks for addressing this issue!

ahuang98 · 2024-11-18T17:41:22Z

core/src/main/scala/kafka/tools/StorageTool.scala

@@ -143,6 +143,9 @@ object StorageTool extends Logging {
    if (namespace.getBoolean("standalone")) {
      formatter.setInitialControllers(createStandaloneDynamicVoters(config))
    }
+    if (namespace.getBoolean("no_initial_controllers")) {
+      formatter.setNoInitialControllers(true)
+    }


nit: can we change the if statement below into an else clause?

ahuang98 · 2024-11-18T17:49:41Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

@@ -130,6 +132,7 @@ public class Formatter {
     * The initial KIP-853 voters.
     */
    private Optional<DynamicVoters> initialControllers = Optional.empty();
+    private boolean noInitialControllers = false;


nit: thoughts on calling this noInitialControllersFlag? might make it more clear this is not just set when initial-controllers is not supplied

FrankYang0529 · 2024-11-19T13:38:28Z

Hi @ahuang98, thanks for the review. I address all comments.

ahuang98 · 2024-11-19T19:56:53Z

@FrankYang0529 thank you! Last two requests, I was wondering if you could add the sequence of events/commands for reproducing the issue to the PR description, and also test your change manually (bring up kraft quorum locally)?

FrankYang0529 · 2024-11-20T15:15:12Z

Hi @ahuang98, thanks for the review. I will update description and test manually tomorrow.

…should be 1 rather than 0 Signed-off-by: PoAn Yang <[email protected]>

FrankYang0529 · 2024-11-21T10:00:33Z

Hi @ahuang98, I updated PR description and following is manually test results:

Setup three controller properties.

# path: ./config/kraft/controller1.properties
process.roles=controller
node.id=1
controller.quorum.bootstrap.servers=localhost:9093
listeners=CONTROLLER://:9093
advertised.listeners=CONTROLLER://localhost:9093
controller.listener.names=CONTROLLER
log.dirs=/tmp/kraft-controller-logs-1

# path: ./config/kraft/controller2.properties
process.roles=controller
node.id=2
controller.quorum.bootstrap.servers=localhost:9093
listeners=CONTROLLER://:9094
advertised.listeners=CONTROLLER://localhost:9094
controller.listener.names=CONTROLLER
log.dirs=/tmp/kraft-controller-logs-2

# path: ./config/kraft/controller3.properties
process.roles=controller
node.id=3
controller.quorum.bootstrap.servers=localhost:9093
listeners=CONTROLLER://:9095
advertised.listeners=CONTROLLER://localhost:9095
controller.listener.names=CONTROLLER
log.dirs=/tmp/kraft-controller-logs-3

Format node 1 as first standalone controller and start it.

> ./bin/kafka-storage.sh format -t "zuV6tFqAQhSGfFe97O-9cw" --config ./config/kraft/controller1.properties --standalone
Formatting dynamic metadata voter directory /tmp/kraft-controller-logs-1 with metadata.version 4.0-IV0.
> ./bin/kafka-server-start.sh ./config/kraft/controller1.properties

Format node 2 with no initial controller and start it.

> ./bin/kafka-storage.sh format -t "zuV6tFqAQhSGfFe97O-9cw" --config ./config/kraft/controller2.properties --no-initial-controllers --feature kraft.version=1
Formatting metadata directory /tmp/kraft-controller-logs-2 with metadata.version 4.0-IV0.
> ./bin/kafka-server-start.sh ./config/kraft/controller2.properties

Format node 3 with no initial controller and start it.

> ./bin/kafka-storage.sh format -t "zuV6tFqAQhSGfFe97O-9cw" --config ./config/kraft/controller3.properties --no-initial-controllers --feature kraft.version=1
Formatting metadata directory /tmp/kraft-controller-logs-3 with metadata.version 4.0-IV0.
> ./bin/kafka-server-start.sh ./config/kraft/controller3.properties

Check cluster status is 1 controller with 2 observers.

> ./bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status
ClusterId:              zuV6tFqAQhSGfFe97O-9cw
LeaderId:               1
LeaderEpoch:            1
HighWatermark:          817
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 1, "directoryId": "zD1nHI9OSyCGu29l3db2yg", "endpoints": ["CONTROLLER://localhost:9093"]}]
CurrentObservers:       [{"id": 2, "directoryId": "vX3tLI_VJKRLAMqJdFUY4Q"}, {"id": 3, "directoryId": "Mk1bbuDgirHiGzOnH4okew"}]

Add node 2 as controller and check cluster status is 2 controllers and 1 observer.

> ./bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 --command-config ./config/kraft/controller2.properties add-controller
Added controller 2 with directory id vX3tLI_VJKRLAMqJdFUY4Q and endpoints: CONTROLLER://localhost:9094
> ./bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status
ClusterId:              zuV6tFqAQhSGfFe97O-9cw
LeaderId:               1
LeaderEpoch:            1
HighWatermark:          960
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 1, "directoryId": "zD1nHI9OSyCGu29l3db2yg", "endpoints": ["CONTROLLER://localhost:9093"]}, {"id": 2, "directoryId": "vX3tLI_VJKRLAMqJdFUY4Q", "endpoints": ["CONTROLLER://localhost:9094"]}]
CurrentObservers:       [{"id": 3, "directoryId": "Mk1bbuDgirHiGzOnH4okew"}]

Add node 3 as controller and check cluster status is 3 controllers and 0 observer.

> ./bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 --command-config ./config/kraft/controller3.properties add-controller
Added controller 3 with directory id Mk1bbuDgirHiGzOnH4okew and endpoints: CONTROLLER://localhost:9095
> ./bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 describe --status
ClusterId:              zuV6tFqAQhSGfFe97O-9cw
LeaderId:               1
LeaderEpoch:            1
HighWatermark:          1123
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [{"id": 1, "directoryId": "zD1nHI9OSyCGu29l3db2yg", "endpoints": ["CONTROLLER://localhost:9093"]}, {"id": 2, "directoryId": "vX3tLI_VJKRLAMqJdFUY4Q", "endpoints": ["CONTROLLER://localhost:9094"]}, {"id": 3, "directoryId": "Mk1bbuDgirHiGzOnH4okew", "endpoints": ["CONTROLLER://localhost:9095"]}]
CurrentObservers:       []

FrankYang0529 · 2024-11-21T10:23:06Z

BTW, I found the adding controller command is different from the KIP-853. Create a followup Jira: https://issues.apache.org/jira/browse/KAFKA-18059.

chia7712 · 2024-11-21T11:02:16Z

I found the adding controller command is different from the KIP-853. Create a followup Jira:

nice find and I have added feedback on the jira

github-actions bot added core Kafka Broker kraft labels Nov 17, 2024

ahuang98 suggested changes Nov 18, 2024

View reviewed changes

FrankYang0529 force-pushed the KAFKA-18028 branch from 4c03c5f to 427381d Compare November 19, 2024 13:37

KAFKA-18028: the effective kraft version of --no-initial-controllers …

847ad3a

…should be 1 rather than 0 Signed-off-by: PoAn Yang <[email protected]>

FrankYang0529 force-pushed the KAFKA-18028 branch from 427381d to 847ad3a Compare November 21, 2024 10:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18028: the effective kraft version of --no-initial-controllers should be 1 rather than 0 #17836

KAFKA-18028: the effective kraft version of --no-initial-controllers should be 1 rather than 0 #17836

FrankYang0529 commented Nov 17, 2024 •

edited

Loading

ahuang98 left a comment

ahuang98 Nov 18, 2024

ahuang98 Nov 18, 2024

FrankYang0529 commented Nov 19, 2024

ahuang98 commented Nov 19, 2024

FrankYang0529 commented Nov 20, 2024

FrankYang0529 commented Nov 21, 2024

FrankYang0529 commented Nov 21, 2024 •

edited

Loading

chia7712 commented Nov 21, 2024

KAFKA-18028: the effective kraft version of --no-initial-controllers should be 1 rather than 0 #17836

Are you sure you want to change the base?

KAFKA-18028: the effective kraft version of --no-initial-controllers should be 1 rather than 0 #17836

Conversation

FrankYang0529 commented Nov 17, 2024 • edited Loading

Committer Checklist (excluded from commit message)

ahuang98 left a comment

Choose a reason for hiding this comment

ahuang98 Nov 18, 2024

Choose a reason for hiding this comment

ahuang98 Nov 18, 2024

Choose a reason for hiding this comment

FrankYang0529 commented Nov 19, 2024

ahuang98 commented Nov 19, 2024

FrankYang0529 commented Nov 20, 2024

FrankYang0529 commented Nov 21, 2024

FrankYang0529 commented Nov 21, 2024 • edited Loading

chia7712 commented Nov 21, 2024

FrankYang0529 commented Nov 17, 2024 •

edited

Loading

FrankYang0529 commented Nov 21, 2024 •

edited

Loading