From 2098c413084d98d5807af82f26c29c010ebb47d4 Mon Sep 17 00:00:00 2001 From: David Arthur Date: Wed, 15 Feb 2023 11:34:38 -0500 Subject: [PATCH 1/4] Add ZK migration docs to the packaged docs --- docs/ops.html | 195 +++++++++++++++++++++++++++++++++++++++++++++++++- docs/toc.html | 1 + 2 files changed, 195 insertions(+), 1 deletion(-) diff --git a/docs/ops.html b/docs/ops.html index 96ea6d0e71a67..922b25b4d8d48 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -3553,6 +3553,7 @@

Kafka server's process.role should be set to either broker or controller but not both. Combined mode can be used in development enviroment but it should be avoided in critical deployment evironments.
  • For redundancy, a Kafka cluster should use 3 controllers. More than 3 servers is not recommended in critical environments. In the rare case of a partial network failure it is possible for the cluster metadata quorum to become unavailable. This limitation will be addresses in a future release of Kafka.
  • The Kafka controllers store all of the metadata for the cluster in memory and on disk. We believe that for a typical Kafka cluster 5GB of main memory and 5GB of disk space on the metadata log director is sufficient.
    • +

    Missing Features

    @@ -3563,9 +3564,201 @@

    Supporting JBOD configurations with multiple storage directories
  • Modifying certain dynamic configurations on the standalone KRaft controller
  • Delegation tokens
    • -
  • Upgrade from ZooKeeper mode
    • +

    ZooKeeper to KRaft Migration

    + +

    + The ZooKeeper to KRaft migration feature is considered Early Access in 3.4.0. It is not recommended for production clusters. +

    + +

    The following features are not yet supported for ZK to KRaft migrations:

    + + + +

    + Please report issues with ZooKeeper to KRaft migration using the + project JIRA and the "kraft" component. +

    + +

    Terminology

    +

    + In this documentation, we use the term "migration" to refer to the process to changing a Kafka cluster's metadata + system from ZooKeeper to KRaft. An "upgrade" refers to installing a newer version of Kafka. It is not recommended to + upgrade the software at the same time as performing a migration. +

    + +

    + This documentation uses the term "ZK mode" to refer to Kafka brokers which are using ZooKeeper as their metadata + system. "KRaft mode" refers Kafka brokers which are using a KRaft controller quorum as their metadata system. +

    + +

    Preparing for migration

    +

    + Before beginning the migration, the Kafka brokers must be upgraded to software version 3.4.0 and have the + "inter.broker.protocol.version" configuration set to "3.4". See Upgrading to 3.4.0 for + upgrade instructions. +

    + +

    + It is recommended to enable TRACE level logging for the migration components while the migration is active. This can + be done by adding the following log4j configuration to each KRaft controller's "log4j.properties" file. +

    + + 
    log4j.logger.org.apache.kafka.metadata.migration=TRACE
    + +

    + It may generally useful to enable DEBUG logging on the KRaft controllers and the ZK brokers during the migration. +

    + +

    Provisioning the KRaft controller quorum

    +

    + Two things are needed before the migration can begin. The brokers must be configured to support the migration and + a KRaft controller quorum must be deployed. The KRaft controllers should be provisioned with the same cluster ID as + the existing Kafka cluster. This can be found by examining one of the "meta.properties" files in the data directories + of the brokers, or by running the following command. +

    + + 
    ./bin/zookeeper-shell.sh localhost:2181 get /cluster/id
    + +

    + The KRaft controller quorum should also be provisioned with the latest metadata.version of "3.4". + For further instructions on KRaft deployment, please refer to the above documentation. +

    + +

    + In addition to the standard KRaft configuration, the KRaft controllers will need to enable support for the migration + as well as provide ZooKeeper connection configuration. +

    + + 
    +# Sample KRaft cluster controller.properties listening on 9093
+process.roles=controller
+node.id=3000
+controller.quorum.voters=3000@localhost:9093
+controller.listener.names=CONTROLLER
+listeners=CONTROLLER://:9093
+
+# Enable the migration
+zookeeper.metadata.migration.enable=true
+
+# ZooKeeper client configuration
+zookeeper.connect=localhost:2181
+
+# Other configs ...
    + +

    Note: The KRaft cluster node.id values must be different from any existing ZK broker broker.id. + In KRaft-mode, the brokers and controllers share the same Node ID namespace.

    + +

    Enabling the migration on the brokers

    +

    + Once the KRaft controller quorum has been started, the brokers will need to be reconfigured and restarted. Brokers + may be restarted in a rolling fashion to avoid impacting cluster availability. Each broker will need to add the + following configurations to allow it to communicate with the KRaft controllers and to enable the migration. +

    + + + +

    Here is a sample config for a broker that is ready for migration:

    + + 
    +# Sample ZK broker server.properties listening on 9092
+broker.id=0
+listeners=PLAINTEXT://:9092
+advertised.listeners=PLAINTEXT://localhost:9092
+listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+
+# Set the IBP
+inter.broker.protocol.version=3.4
+
+# Enable the migration
+zookeeper.metadata.migration.enable=true
+
+# ZooKeeper client configuration
+zookeeper.connect=localhost:2181
+
+# KRaft controller quorum configuration
+controller.quorum.voters=3000@localhost:9093
+controller.listener.names=CONTROLLER
    + +

    + Note: Once the final ZK broker has been restarted with the necessary configuration, the migration will automatically begin. + When the migration is complete, a INFO level log can be observed on the active controller. +

    + + 
    Completed migration of metadata from Zookeeper to KRaft
    + +

    Migrating brokers to KRaft

    +

    + Once the KRaft controller completes the metadata migration, the brokers will still be running in ZK mode. While the + KRaft controller is in migration mode, it will continue sending controller RPCs to the ZK mode brokers. This includes + RPCs like UpdateMetadata and LeaderAndIsr. +

    + +

    + To migrate the brokers to KRaft, they simply need to be reconfigured as KRaft brokers and restarted. Using the above + broker configuration as an example, we would replace the broker.id with node.id and add + process.roles=broker. It is important that the broker maintain the same Broker/Node ID when it is restarted. + The zookeeper configurations should be removed at this point. +

    + + 
    +# Sample KRaft broker server.properties listening on 9092
+process.roles=broker
+node.id=0
+listeners=PLAINTEXT://:9092
+advertised.listeners=PLAINTEXT://localhost:9092
+listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
+
+# Don't set the IBP, KRaft uses "metadata.version" feature flag
+# inter.broker.protocol.version=3.4
+
+# Remove the migration enabled flag
+# zookeeper.metadata.migration.enable=true
+
+# Remove ZooKeeper client configuration
+# zookeeper.connect=localhost:2181
+
+# Keep the KRaft controller quorum configuration
+controller.quorum.voters=3000@localhost:9093
+controller.listener.names=CONTROLLER
    + +

    + Each broker is restarted with a KRaft configuration until the entire cluster is running in KRaft mode. +

    + +

    Finalizing the migration

    +

    + Once all brokers have been restarted in KRaft mode, the last step to finalize the migration is to take the + KRaft controllers out of migration mode. This is done by removing the "zookeeper.metadata.migration.enable" + property from each of their configs and restarting them one at a time. +

    + + 
    +# Sample KRaft cluster controller.properties listening on 9093
+process.roles=controller
+node.id=3000
+controller.quorum.voters=1@localhost:9093
+controller.listener.names=CONTROLLER
+listeners=CONTROLLER://:9093
+
+# Disable the migration
+# zookeeper.metadata.migration.enable=true
+
+# Remove ZooKeeper client configuration
+# zookeeper.connect=localhost:2181
+
+# Other configs ...
    +
    diff --git a/docs/toc.html b/docs/toc.html index 356bf52f4315b..aad286b25925e 100644 --- a/docs/toc.html +++ b/docs/toc.html @@ -161,6 +161,7 @@
  • Debugging
  • Deploying Considerations
  • Missing Features
    • +
  • ZooKeeper to KRaft Migration
    • From f0127a7c6c7bab1f51592dcef9773659f7921d77 Mon Sep 17 00:00:00 2001 From: David Arthur Date: Wed, 15 Feb 2023 11:37:09 -0500 Subject: [PATCH 2/4] Bring in latest changes --- docs/ops.html | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/docs/ops.html b/docs/ops.html index 922b25b4d8d48..c24a99e99381e 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -3550,14 +3550,14 @@
    Deploying Considerations

    Missing Features

    -

    The following features are not fullying implemented in KRaft mode:

    +

    The following features are not fully implemented in KRaft mode: