Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export numerical kafka_topic_info parameters/labels as individual metrics #116

Open
amuraru opened this issue Sep 20, 2021 · 1 comment
Open

Comments

@amuraru
Copy link
Contributor

amuraru commented Sep 20, 2021

Currently kminion_kafka_topic_info metric exposes all topic parameters as labels. [1]

I've got an use-case where I'd need the replication_factor and min_insync_replicas values to be used in promql queries to verify if a given topic has under-min replicas, e.g:
kafka_topic_partition_replicas < "min_insync_replicas"
or a given topic has under-replicated partitions:
kafka_topic_partition_replicas < "replication_factor"

To do so, the only solution I found is to have these numerical topic parameters exposed as separate metrics, e.g:
kminion_kafka_topic_info_min_insync_replicas
kminion_kafka_topic_info_replication_factor

Would this be a good improvement for the kminion @weeco ?

[1] Sample:

kminion_kafka_topic_info{cleanup_policy="delete", container="kafka-minion", endpoint="metrics", 
max_message_bytes="1048588", message_format_version="2.5-IV0", message_timestamp_type="CreateTime", 
min_insync_replicas="2", partition_count="1",  replication_factor="1", retention_bytes="-1", retention_ms="10000", 
segment_bytes="536870912",
topic_name="00faec4f386fc8dda9688d402cc92d0c1082ee0d426909841e6b4d11750deb65"}
@amuraru amuraru changed the title Export numerical kafka_topic_info configurations as individual metrics Export numerical kafka_topic_info parameters/labels as individual metrics Sep 20, 2021
@weeco
Copy link
Contributor

weeco commented Sep 20, 2021

I think Kafka itself has a metric called under min isr. I think introducing a similiar metric would make sense. Literally you could expose via a metric how many under.min.isr.partitions exist on a given topic:

Example:

kminion_kafka_topic_under_min_isr_partitions{topic_name="xy"} 4

WDYT?

amuraru added a commit to amuraru/kminion that referenced this issue Sep 20, 2021
…ndividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms


Partial fix for redpanda-data#116
amuraru added a commit to amuraru/kminion that referenced this issue Oct 23, 2021
…ndividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms


Partial fix for redpanda-data#116
amuraru added a commit to amuraru/kminion that referenced this issue Oct 23, 2021
…ndividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms


Partial fix for redpanda-data#116
amuraru added a commit to amuraru/kminion that referenced this issue Oct 24, 2021
…ndividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms


Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Dec 22, 2021
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Dec 22, 2021
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Dec 22, 2021
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue May 5, 2022
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Jun 27, 2022
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Jun 28, 2022
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue Dec 16, 2022
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue May 5, 2023
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue May 5, 2023
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
amuraru added a commit to adobe/kminion that referenced this issue May 14, 2023
…ividual metrics

Export as numerical metrics:

- topic_info_partitions_count
- topic_info_replication_factor
- topic_info_min_insync_replicas
- topic_info_retention_ms

Partial fix for redpanda-data#116
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants