Skip to content

MINOR: add missing docs for record-e2e-latency metrics #10251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ableegoldman merged 7 commits into from
Mar 4, 2021
Merged

MINOR: add missing docs for record-e2e-latency metrics #10251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a4c9250
add missing metrics
ableegoldman Mar 3, 2021
a95d25e
add missing active-process-ratio
ableegoldman Mar 3, 2021
eb522d7
standardize formatting
ableegoldman Mar 3, 2021
0dccb07
fix typo in release script on the side
ableegoldman Mar 3, 2021
d306f53
fix typo, break up long lines
ableegoldman Mar 4, 2021
bf7f8d7
add TRACE level
ableegoldman Mar 4, 2021
e7e3229
fix name
ableegoldman Mar 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 41 additions & 5 deletions docs/ops.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2108,7 +2108,7 @@ <h4 class="anchor-heading"><a id="connect_monitoring" class="anchor-link"></a><a
<h4 class="anchor-heading"><a id="kafka_streams_monitoring" class="anchor-link"></a><a href="#kafka_streams_monitoring">Streams Monitoring</a></h4>

A Kafka Streams instance contains all the producer and consumer metrics as well as additional metrics specific to Streams.
By default Kafka Streams has metrics with two recording levels: <code>debug</code> and <code>info</code>.
By default Kafka Streams has metrics with three recording levels: <code>info</code>, <code>debug</code>, and <code>trace</code>.

<p>
Note that the metrics have a 4-layer hierarchy. At the top level there are client-level metrics for each started
Expand Down Expand Up @@ -2277,8 +2277,8 @@ <h5 class="anchor-heading"><a id="kafka_streams_thread_monitoring" class="anchor
</table>

<h5 class="anchor-heading"><a id="kafka_streams_task_monitoring" class="anchor-link"></a><a href="#kafka_streams_task_monitoring">Task Metrics</a></h5>
All of the following metrics have a recording level of <code>debug</code>, except for metrics
dropped-records-rate and dropped-records-total which have a recording level of <code>info</code>:
All of the following metrics have a recording level of <code>debug</code>, except for the dropped-records-*,
active-process-ratio, and record-e2e-latency-* metrics which have a recording level of <code>info</code>:
<table class="data-table">
<tbody>
<tr>
Expand Down Expand Up @@ -2356,6 +2356,26 @@ <h5 class="anchor-heading"><a id="kafka_streams_task_monitoring" class="anchor-l
<td>The total number of records dropped within this task.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>active-process-ratio</td>

@ableegoldman ableegoldman Mar 3, 2021

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed that we seem to be missing this task-level metric as well. Might be good to take stock of all the metrics at some point and make sure they're documented clearly, especially since it's easy to miss this doc which is outside of the usual Streams docs

<td>The total number of records dropped within this task.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-avg</td>
<td>The average end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-max</td>
<td>The maximum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-min</td>
<td>The minimum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-task-metrics,thread-id=([-.\w]+),task-id=([-.\w]+)</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -2394,8 +2414,9 @@ <h5 class="anchor-heading"><a id="kafka_streams_node_monitoring" class="anchor-l
</table>

<h5 class="anchor-heading"><a id="kafka_streams_store_monitoring" class="anchor-link"></a><a href="#kafka_streams_store_monitoring">State Store Metrics</a></h5>
All of the following metrics have a recording level of <code>debug</code>. Note that the <code>store-scope</code> value is specified in <code>StoreSupplier#metricsScope()</code> for user's customized
state stores; for built-in state stores, currently we have:
All of the following metrics have a recording level of <code>debug</code>, except for the record-e2e-latency-* metrics which have a recording level <code>trace></code>.
Note that the <code>store-scope</code> value is specified in <code>StoreSupplier#metricsScope()</code> for user's customized state stores;
for built-in state stores, currently we have:
<ul>
<li><code>in-memory-state</code></li>
<li><code>in-memory-lru-state</code></li>
Expand Down Expand Up @@ -2570,6 +2591,21 @@ <h5 class="anchor-heading"><a id="kafka_streams_store_monitoring" class="anchor-
<td>The maximum number of records buffered over the sampling window.</td>
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),in-memory-suppression-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-avg</td>
<td>The average end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-max</td>
<td>The maximum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
</tr>
<tr>
<td>record-e2e-latency-min</td>
<td>The minimum end-to-end latency of a record, measured by comparing the record timestamp with the system time when it has been fully processed by the node.</td>
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
</tr>
</tbody>
</table>

Expand Down
2 changes: 1 addition & 1 deletion release.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,7 @@ def command_release_announcement_email():
</server>
<server>
<id>your-gpgkeyId</id>
<passphrase>your-gpg-passphase</passphrase>
<passphrase>your-gpg-passphrase</passphrase>
</server>
<profile>
<id>gpg-signing</id>
Expand Down