Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add node name in node-exporter metrics #4018

Open
gdemonet opened this issue Mar 8, 2023 · 0 comments
Open

Add node name in node-exporter metrics #4018

gdemonet opened this issue Mar 8, 2023 · 0 comments
Labels
complexity:medium Something that requires one or few days to fix topic:monitoring Everything related to monitoring of services in a running cluster

Comments

@gdemonet
Copy link
Contributor

gdemonet commented Mar 8, 2023

Component: node-exporter, prometheus

Why this is needed:

  • Enable using node-exporter metrics in prometheus-adapter (see Bump kube-prometheus-stack to 45.5.0 #4017 (comment))
  • Exploring node-exporter metrics (e.g. using Grafana Explore tab) would be easier
  • Querying these metrics from our UI would avoid retrieving node IPs from Salt (to build the "instance" label as $control_plane_ip:$node_exporter_port)

What should be done:

Current service discovery for node-exporter is only using the Service (so, Endpoints), hence, we don't get node metadata. This SO answer seems to point to a solution (i.e. use kubernetes_sd_configs with node role: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#node).
@gdemonet gdemonet added complexity:medium Something that requires one or few days to fix topic:monitoring Everything related to monitoring of services in a running cluster labels Mar 8, 2023
@gdemonet gdemonet mentioned this issue Mar 8, 2023
Merged
gdemonet added a commit that referenced this issue Mar 8, 2023
@gdemonet
charts,salt: Reinstate node-level cAdvisor metrics
97ce7c6 
This changes the default configuration from kube-prometheus-stack since
we still use these metrics in prometheus-adapter.
Ideally, we would rather let prometheus-adapter consume node-exporter
metrics, but this requires #4018 to be fixed first.
gdemonet added a commit that referenced this issue Mar 9, 2023
@gdemonet
charts,salt: Reinstate node-level cAdvisor metrics
3564da7 
This changes the default configuration from kube-prometheus-stack since
we still use these metrics in prometheus-adapter.
Ideally, we would rather let prometheus-adapter consume node-exporter
metrics, but this requires #4018 to be fixed first.
gdemonet added a commit that referenced this issue Mar 9, 2023
@gdemonet
charts,salt: Reinstate node-level cAdvisor metrics
b9d1e26 
This changes the default configuration from kube-prometheus-stack since
we still use these metrics in prometheus-adapter.
Ideally, we would rather let prometheus-adapter consume node-exporter
metrics, but this requires #4018 to be fixed first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
complexity:medium Something that requires one or few days to fix topic:monitoring Everything related to monitoring of services in a running cluster
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gdemonet