Skip to content

KAFKA-14205; Document how to replace the disk for the KRaft Controller #12597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jsancio merged 4 commits into from
Sep 12, 2022
Merged

KAFKA-14205; Document how to replace the disk for the KRaft Controller #12597

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8fd3221
KAFKA-14205; Document how to replace the disk for the KRaft Controller
jsancio Sep 6, 2022
cb2d864
Fixed some of the wording for this change
jsancio Sep 9, 2022
16b8b24
Fix linked for replace disk section
jsancio Sep 9, 2022
b96cdbe
More wording changes
jsancio Sep 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/ops.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1373,6 +1373,27 @@ <h5 class="anchor-heading"><a id="ext4" class="anchor-link"></a><a href="#ext4">
<li>delalloc: Delayed allocation means that the filesystem avoid allocating any blocks until the physical write occurs. This allows ext4 to allocate a large extent instead of smaller pages and helps ensure the data is written sequentially. This feature is great for throughput. It does seem to involve some locking in the filesystem which adds a bit of latency variance.
</ul>

<h4 class="anchor-heading"><a id="replace_disk" class="anchor-link"></a><a href="#replaced_disk">Replace KRaft Controller Disk</a></h4>
<p>When Kafka is configured to use KRaft instead of ZooKeeper, the controllers stores the cluster metadata in the directory specified in <code>metadata.log.dir</code>, <code>log.dir</code> or <code>log.dirs</code>. See the documentation for <code>metadata.log.dir</code> for details.</p>
Comment thread
jsancio marked this conversation as resolved.
Outdated

<p>If the data in the cluster metdata directory (disk) is lost either because of hardware failure or the hardware needs to be replace, care should be taken when provisioning the new controller node. The new controller node should not be formatted and started until the majority of the controllers have all of the committed data. To determine if the majority of the controllers have the committed data, run the <code>kafka-metadata-quorum.sh</code> tool to describe the replication status:
Comment thread
jsancio marked this conversation as resolved.
Outdated

<pre class="line-numbers"><code class="language-bash"> &gt; bin/kafka-metadata-quorum.sh --bootstrap-server broker_host:port describe --replication
NodeId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp Status
1 25806 0 1662500992757 1662500992757 Leader
... ... ... ... ... ...
</code></pre>

Check and wait until the <code>Lag</code> is small for the majority of the controllers. Check and wait until the <code>LastFetchTimestamp</code> and <code>LastCaughtUpTimestamp</code> are close to each other for the majority of the controllers. At this point it is safer to format the controller's metadata log directory. This can be done by running the <code>kafka-storage.sh</code> command.
Comment thread
jsancio marked this conversation as resolved.
Outdated

<pre class="line-numbers"><code class="language-bash"> &gt; bin/kafka-storage.sh format --cluster-id uuid --config server_properties</code></pre>

<p>If multiple log directories and metadata directory are used but only one of them is getting replaced it may be necessary to run <code>kafka-storage.sh format</code> with the <code>--ignore-formatted</code> option.</p>
Comment thread
jsancio marked this conversation as resolved.
Outdated

<p>Start the the KRaft controller after formatting the log directories.</p>

<pre class="line-numbers"><code class="language-bash"> &gt; /bin/kafka-server-start.sh server_properties</code></pre>

<h3 class="anchor-heading"><a id="monitoring" class="anchor-link"></a><a href="#monitoring">6.8 Monitoring</a></h3>

Kafka uses Yammer Metrics for metrics reporting in the server. The Java clients use Kafka Metrics, a built-in metrics registry that minimizes transitive dependencies pulled into client applications. Both expose metrics via JMX and can be configured to report stats using pluggable stats reporters to hook up to your monitoring system.
Expand Down