Skip to content

Commit 5982e2c

Browse files
committed
Fix some typos and reword a few things
1 parent 10ed356 commit 5982e2c

File tree

1 file changed

+31
-26
lines changed

1 file changed

+31
-26
lines changed

docs/end-to-end.md

+31-26
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,61 @@
11
# End-To-End Monitoring
22

3-
This page describes the end-to-end monitoring feature in kminion, how it works, and what metrics it provides.
3+
This page describes the end-to-end monitoring feature in KMinion, how it works, and what metrics it provides.
44

55
## Motivation
66
> What is the issue? Why did we build this feature?
77
8-
We can (and do!) monitor metrics like CPU usage, free disk space, or even consumer group lag.
9-
However those metrics don't give us a good idea of the performance characteristics an actual, real-world, client can expect from the cluster.
8+
We can monitor metrics like CPU usage, free disk space, or even consumer group lag.
9+
However, these metrics don't give us a good idea of the performance characteristics an actual, real-world, client
10+
experiences when connected to the cluster.
1011

1112
With the "classic" metrics lots of questions go unanswered:
12-
- Can a client even produce messages to the cluster in general?
13-
- Does the cluster still give a an acceptable performance for producing messages, committing offsets, and consuming messages?
14-
- What if only a subset of the brokers is somehow unable to handle requests?
13+
- Can a client produce messages to the cluster?
14+
- Can clients produce & consume messages as well as commit group offsets with an acceptable latency?
15+
- Is the cluster in a healthy state from a client's perspective?
1516

1617
## Approach & Implementation
1718
> How do we solve those issues? How does the feature work?
1819
19-
The most reliably way to get real-world performance and availability metrics is to just actually run a producer/consumer ourselves - which is exactly what the end-to-end monitoring feature does!
20+
The most reliably way to get real-world performance and availability metrics is to actually run a producer/consumer
21+
ourselves. This is exactly what the end-to-end monitoring feature does!
2022

2123
## High Level Overview
22-
In order to determine if the cluster is fully operational and its performance is within acceptable limits, kminion continously produces and consumes messages to/from the cluster. That way we can measure things like ack-latency, commit-latency, and roundtrip-time.
24+
In order to determine if the cluster is fully operational, and it's performance is within acceptable limits,
25+
KMinion continuously produces and consumes messages to/from the cluster. That way we can measure things like ack-latency,
26+
commit-latency, and roundtrip-time.
2327

24-
Kminion creates and manages its own topic for the end-to-end test messages. The name of the topic can be configured.
28+
KMinion creates and manages its own topic for the end-to-end test messages. The name of the topic can be configured.
2529

2630
**The first step** is to create a message and send it to the cluster.
27-
- Every produced message is added to an internal tracker so we can recognize messages being "lost" (messages that don't arrive back at the consumer within the configured time span).
31+
- Every produced message is added to an internal tracker, so we can recognize messages being "lost".
32+
A message is considered lost if it doesn't arrive back at the consumer within the configured time span.
2833

29-
**The second step** is to continously consume the topic.
30-
- As each message arrives, we calculate its roundtrip time (time from the point the message was created, until kminion received it again)
34+
**The second step** is to continuously consume the topic.
35+
- As each message arrives, we calculate its roundtrip time (time from the point the message was created, until KMinion received it again)
3136
- Consumer group offsets are committed periodically, while also recording the time each commit takes.
3237

3338
### Topic Management
34-
The topic kminion uses is created and managed completely automatically (the topic name can be configured though).
39+
The topic KMinion uses, is created and managed completely automatically (the topic name can be configured though).
3540

36-
Kminion continously verifies the topic and fixes issues automatically:
37-
- Add partitions to the topic so it has at least as many partitions as there are brokers.
38-
- Will reassign partitions to ensure every broker leads at least one partition, and that all partitions' replicas are distributed evenly across the brokers. (Preferably assigns so partitionID and leader brokerID matches)
41+
KMinion continuously checks the topic and fixes issues/imbalances automatically:
42+
- Add partitions to the topic, so it has at least as many partitions as there are brokers.
43+
- Will reassign partitions to ensure every broker leads at least one partition, and that all partitions' replicas
44+
are distributed evenly across the brokers. KMinion tries to assign partitionIDs to brokers that have the same broker id.
3945

4046

4147
### Consumer Group Management
42-
On startup each kminion instance generates a unique identifier (UUID) used to create its own consumer group - using the shared prefix from the config.
48+
On startup each KMinion instance generates a unique identifier (UUID) that is used to create its own consumer group.
49+
It incorporates the shared prefix from the config.
4350

44-
That is neccesary because:
51+
That is necessary because:
4552
- Offsets must not be shared among multiple instances.
4653
- Each instance must always consume **all** partitions of the topic.
4754

4855
The instances' UUID is also embedded in every message, so each instance can easily filter out messages it didn't produce.
49-
That's why it is perfectly fine to run multiple kminion instances against the same cluster, using the same topic.
56+
That's why it is perfectly fine to run multiple KMinion instances against the same cluster, using the same topic.
5057

51-
Kminion also monitors and deletes consumer groups that use its configured prefix.
58+
KMinion also monitors and deletes consumer groups that use it's configured prefix.
5259
That way, when an instance exits/restarts, previous consumer groups will be cleaned up quickly (check happens every 20s).
5360

5461

@@ -58,7 +65,7 @@ The end-to-end monitoring feature exports the following metrics.
5865
### Counters
5966
| Name | Description |
6067
| --- | --- |
61-
| `kminion_end_to_end_messages_produced_total ` | Messages kminion *tried* to send |
68+
| `kminion_end_to_end_messages_produced_total ` | Messages KMinion *tried* to send |
6269
| `kminion_end_to_end_messages_acked_total ` | Messages actually sent and acknowledged by the cluster |
6370
| `kminion_end_to_end_messages_received_total ` | Number of messages received (only counts those that match, i.e. that this instance actually produced itself) |
6471
| `kminion_end_to_end_commits_total` | Number of successful offset commits |
@@ -67,11 +74,9 @@ The end-to-end monitoring feature exports the following metrics.
6774
### Histograms
6875
| Name | Description |
6976
| --- | --- |
70-
| `kminion_end_to_end_produce_latency_seconds ` | Duration until a the cluster acknowledged a message. |
77+
| `kminion_end_to_end_produce_latency_seconds ` | Duration until the cluster acknowledged a message. |
7178
| `kminion_end_to_end_commit_latency_seconds` | Duration of offset commits. Has a label for coordinator brokerID that answered the commit request |
72-
| `kminion_end_to_end_roundtrip_latency_seconds ` | Duration from creation of a message, until it was received/consumed agagin. |
73-
74-
79+
| `kminion_end_to_end_roundtrip_latency_seconds ` | Duration from creation of a message, until it was received/consumed again. |
7580

7681
## Config Properties
7782
All config properties related to this feature are located in `minion.endToEnd`.
@@ -83,7 +88,7 @@ All config properties related to this feature are located in `minion.endToEnd`.
8388
topicManagement:
8489
# You can disable topic management, without disabling the testing feature.
8590
# Only makes sense if you have multiple kminion instances, and for some reason only want one of them to create/configure the topic.
86-
# It is strongly reccommended to leave this enabled.
91+
# It is strongly recommended to leave this enabled.
8792
enabled: true
8893

8994
# Name of the topic kminion uses to send its test messages

0 commit comments

Comments
 (0)