From 8fd3221bdd716259820dd2673d8626d22e3afdfa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Armando=20Garc=C3=ADa=20Sancio?= Date: Tue, 6 Sep 2022 14:52:38 -0700 Subject: [PATCH 1/4] KAFKA-14205; Document how to replace the disk for the KRaft Controller --- docs/ops.html | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/ops.html b/docs/ops.html index 1854cf057c2fc..e98991449d267 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -1373,6 +1373,27 @@
  • delalloc: Delayed allocation means that the filesystem avoid allocating any blocks until the physical write occurs. This allows ext4 to allocate a large extent instead of smaller pages and helps ensure the data is written sequentially. This feature is great for throughput. It does seem to involve some locking in the filesystem which adds a bit of latency variance. +

    Replace KRaft Controller Disk

    +

    When Kafka is configured to use KRaft instead of ZooKeeper, the controllers stores the cluster metadata in the directory specified in metadata.log.dir, log.dir or log.dirs. See the documentation for metadata.log.dir for details.

    + +

    If the data in the cluster metdata directory (disk) is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: + + 

     > bin/kafka-metadata-quorum.sh --bootstrap-server broker_host:port describe --replication
+ NodeId  LogEndOffset    Lag     LastFetchTimestamp      LastCaughtUpTimestamp   Status
+ 1       25806           0       1662500992757           1662500992757           Leader
+ ...     ...             ...     ...                     ...                     ...
+
    + + Check and wait until the Lag is small for the majority of the controllers. Check and wait until the LastFetchTimestamp and LastCaughtUpTimestamp are close to each other for the majority of the controllers. At this point it is safer to format the controller's metadata log directory. This can be done by running the kafka-storage.sh command. + + 
     > bin/kafka-storage.sh format --cluster-id uuid --config server_properties
    + +

    If multiple log directories and metadata directory are used but only one of them is getting replaced it may be necessary to run kafka-storage.sh format with the --ignore-formatted option.

    + +

    Start the the KRaft controller after formatting the log directories.

    + + 
     > /bin/kafka-server-start.sh server_properties
    +

    6.8 Monitoring

    Kafka uses Yammer Metrics for metrics reporting in the server. The Java clients use Kafka Metrics, a built-in metrics registry that minimizes transitive dependencies pulled into client applications. Both expose metrics via JMX and can be configured to report stats using pluggable stats reporters to hook up to your monitoring system. From cb2d86463cb92b5ea49f30eac847d5ce9831a379 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Armando=20Garc=C3=ADa=20Sancio?= Date: Fri, 9 Sep 2022 14:20:37 -0700 Subject: [PATCH 2/4] Fixed some of the wording for this change --- docs/ops.html | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ops.html b/docs/ops.html index e98991449d267..7cc35e23a8cf4 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -1374,9 +1374,9 @@

    Replace KRaft Controller Disk

    -

    When Kafka is configured to use KRaft instead of ZooKeeper, the controllers stores the cluster metadata in the directory specified in metadata.log.dir, log.dir or log.dirs. See the documentation for metadata.log.dir for details.

    +

    When Kafka is configured to use KRaft, the controllers store the cluster metadata in the directory specified in metadata.log.dir -- or the first log directory, if metadata.log.dir is not configured. See the documentation for metadata.log.dir for details.

    -

    If the data in the cluster metdata directory (disk) is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: +

    If the data in the cluster metdata directory is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: 

     > bin/kafka-metadata-quorum.sh --bootstrap-server broker_host:port describe --replication
  NodeId  LogEndOffset    Lag     LastFetchTimestamp      LastCaughtUpTimestamp   Status
@@ -1388,9 +1388,9 @@ 
     > bin/kafka-storage.sh format --cluster-id uuid --config server_properties
    -

    If multiple log directories and metadata directory are used but only one of them is getting replaced it may be necessary to run kafka-storage.sh format with the --ignore-formatted option.

    +

    It is possible for the bin/kafka-storage.sh format command above to fail with a message like Log directory ... is already formatted. This can happend when combined mode is used and only the metadata log directory was lost but not the others. In that case and only in that case, can you run the kafka-storage.sh format command with the --ignore-formatted option.

    -

    Start the the KRaft controller after formatting the log directories.

    +

    Start the KRaft controller after formatting the log directories.

     > /bin/kafka-server-start.sh server_properties
    From 16b8b24129e71acadaa7d1137999fba3de177938 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Armando=20Garc=C3=ADa=20Sancio?= Date: Fri, 9 Sep 2022 14:37:15 -0700 Subject: [PATCH 3/4] Fix linked for replace disk section --- docs/ops.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ops.html b/docs/ops.html index 7cc35e23a8cf4..fecc92b4e09d4 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -1373,7 +1373,7 @@
  • delalloc: Delayed allocation means that the filesystem avoid allocating any blocks until the physical write occurs. This allows ext4 to allocate a large extent instead of smaller pages and helps ensure the data is written sequentially. This feature is great for throughput. It does seem to involve some locking in the filesystem which adds a bit of latency variance. -

    Replace KRaft Controller Disk

    +

    Replace KRaft Controller Disk

    When Kafka is configured to use KRaft, the controllers store the cluster metadata in the directory specified in metadata.log.dir -- or the first log directory, if metadata.log.dir is not configured. See the documentation for metadata.log.dir for details.

    If the data in the cluster metdata directory is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: From b96cdbe0131a598b585d45c15e3121b8e830f4eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Armando=20Garc=C3=ADa=20Sancio?= Date: Mon, 12 Sep 2022 16:55:35 -0700 Subject: [PATCH 4/4] More wording changes --- docs/ops.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ops.html b/docs/ops.html index fecc92b4e09d4..da13ad9b444c1 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -1376,7 +1376,7 @@

    Replace KRaft Controller Disk

    When Kafka is configured to use KRaft, the controllers store the cluster metadata in the directory specified in metadata.log.dir -- or the first log directory, if metadata.log.dir is not configured. See the documentation for metadata.log.dir for details.

    -

    If the data in the cluster metdata directory is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status: +

    If the data in the cluster metdata directory is lost either because of hardware failure or the hardware needs to be replaced, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the kafka-metadata-quorum.sh tool to describe the replication status:

     > bin/kafka-metadata-quorum.sh --bootstrap-server broker_host:port describe --replication
  NodeId  LogEndOffset    Lag     LastFetchTimestamp      LastCaughtUpTimestamp   Status
@@ -1384,7 +1384,7 @@
    - Check and wait until the Lag is small for the majority of the controllers. Check and wait until the LastFetchTimestamp and LastCaughtUpTimestamp are close to each other for the majority of the controllers. At this point it is safer to format the controller's metadata log directory. This can be done by running the kafka-storage.sh command. +

    Check and wait until the Lag is small for a majority of the controllers. If the leader's end offset is not increasing, you can wait until the lag is 0 for a majority; otherwise, you can pick the latest leader end offset and wait until all replicas have reached it. Check and wait until the LastFetchTimestamp and LastCaughtUpTimestamp are close to each other for the majority of the controllers. At this point it is safer to format the controller's metadata log directory. This can be done by running the kafka-storage.sh command.

     > bin/kafka-storage.sh format --cluster-id uuid --config server_properties