Skip to content

KAFKA-10473: Add docs on partition size-on-disk, and other log-related metrics #9276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guozhangwang merged 4 commits into from
Dec 3, 2020
Merged

KAFKA-10473: Add docs on partition size-on-disk, and other log-related metrics #9276

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fd48631
Add docs on the following JMX metrics
Sep 9, 2020
93d93a0
Merge remote-tracking branch 'origin/trunk' into add-to-docs-kafka-lo…
Sep 23, 2020
062c134
Merge remote-tracking branch 'origin/trunk' into add-to-docs-kafka-lo…
Oct 9, 2020
01bd153
Merge remote-tracking branch 'upstream/trunk' into add-to-docs-kafka-…
Dec 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/ops.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
<script id="ops-template" type="text/x-handlebars-template">

Here is some information on actually running Kafka as a production system based on usage and experience at LinkedIn. Please send us any additional tips you know of.

Expand Down Expand Up @@ -1040,6 +1039,26 @@ <h4 class="anchor-heading"><a id="remote_jmx" class="anchor-link"></a><a href="#
<td>kafka.server:type=BrokerTopicMetrics,name=ReassignmentBytesInPerSec</td>
<td></td>
</tr>
<tr>
<td>Size of a partition on disk (in bytes)</td>
<td>kafka.log:type=Log,name=Size,topic=([-.\w]+),partition=([0-9]+)</td>
<td>The size of a partition on disk, measured in bytes.</td>
</tr>
<tr>
<td>Number of log segments in a partition</td>
<td>kafka.log:type=Log,name=NumLogSegments,topic=([-.\w]+),partition=([0-9]+)</td>
<td>The number of log segments in a partition.</td>

@wushujames wushujames Sep 9, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure what information to add for this column of "normal value".

  • For date-based retention topics, it should roughly be sizeOfTopic/segmentSize
  • But for compacted topics, that isn't the case
  • If this is too high, it will mean more file handles.
  • If it is too high, it might inefficient (is this true?)

Anyway, I wasn't sure what to put, so I left it simple.

@guozhangwang guozhangwang Dec 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine :)

</tr>
<tr>
<td>First offset in a partition</td>
<td>kafka.log:type=Log,name=LogStartOffset,topic=([-.\w]+),partition=([0-9]+)</td>
<td>The first offset in a partition.</td>
</tr>
<tr>
<td>Last offset in a partition</td>
<td>kafka.log:type=Log,name=LogEndOffset,topic=([-.\w]+),partition=([0-9]+)</td>
<td>The last offset in a partition.</td>

@wushujames wushujames Sep 9, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the last committed offset? Or, is it uncommitted offsets? Meaning, if a producer appends a record there but the followers haven't yet acked it, will this value get incremented?

Does that level of detail matter, in the docs?

@wushujames wushujames Sep 9, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tag @hachikuji

@guozhangwang guozhangwang Dec 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the last uncomitted offset: after the local append it would be incremented, not waiting for other replicas to ack.

</tr>
</tbody></table>

<h4><a id="selector_monitoring" href="#selector_monitoring">Common monitoring metrics for producer/consumer/connect/streams</a></h4>
Expand Down