Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reason label to total_reload metric #1099

Merged
merged 5 commits into from
Aug 24, 2020
Merged

Add reason label to total_reload metric #1099

merged 5 commits into from
Aug 24, 2020

Conversation

LorcanMcVeigh
Copy link
Contributor

Proposed changes

Add a label reason to the prometheus metric total_reloads which specifies if the reload was caused by a endpoint update or another reason

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation
  • I have rebased my branch onto master
  • I will ensure my PR is targeting the master branch and pulling from my branch from my own fork

Copy link
Contributor

@pleshakov pleshakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LorcanMcVeigh
Please see my comments and suggestions

Additionally, after you start the IC, until it gets reloaded, you will not get metrics for one or both labels. For "other" label, as the first reload happens quickly after the start, it is hard to notice its absence. However, for the "endpoints" label, you can see its absence. For example, here the IC was reloaded but no endpoints update happened yet:

# HELP nginx_ingress_controller_nginx_reloads_total Number of successful NGINX reloads
# TYPE nginx_ingress_controller_nginx_reloads_total counter
nginx_ingress_controller_nginx_reloads_total{class="nginx",reason="other"} 2

Note that the metric nginx_ingress_controller_nginx_reloads_total{class="nginx",reason="endpoint"} is not present here at all.

The solution is to initialize those labels before the IC starts. Could you add that?

docs-web/logging-and-monitoring/prometheus.md Outdated Show resolved Hide resolved
internal/metrics/collectors/manager.go Outdated Show resolved Hide resolved
internal/nginx/manager.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
@LorcanMcVeigh
Copy link
Contributor Author

@LorcanMcVeigh
Please see my comments and suggestions

Additionally, after you start the IC, until it gets reloaded, you will not get metrics for one or both labels. For "other" label, as the first reload happens quickly after the start, it is hard to notice its absence. However, for the "endpoints" label, you can see its absence. For example, here the IC was reloaded but no endpoints update happened yet:

# HELP nginx_ingress_controller_nginx_reloads_total Number of successful NGINX reloads
# TYPE nginx_ingress_controller_nginx_reloads_total counter
nginx_ingress_controller_nginx_reloads_total{class="nginx",reason="other"} 2

Note that the metric nginx_ingress_controller_nginx_reloads_total{class="nginx",reason="endpoint"} is not present here at all.

The solution is to initialize those labels before the IC starts. Could you add that?

could you possibly tell me how you achieved this? I haven't been able to replicate this

@pleshakov
Copy link
Contributor

@LorcanMcVeigh

could you possibly tell me how you achieved this? I haven't been able to replicate this

yep. So just deploy an Ingress Controller. Without deploying any backends and Ingress/VS resources.

deployments/helm-chart/values.yaml Outdated Show resolved Hide resolved
docs-web/logging-and-monitoring/prometheus.md Outdated Show resolved Hide resolved
internal/configs/configurator.go Outdated Show resolved Hide resolved
internal/metrics/collectors/manager.go Outdated Show resolved Hide resolved
internal/nginx/manager.go Outdated Show resolved Hide resolved
LorcanMcVeigh and others added 2 commits August 21, 2020 10:45
Co-authored-by: Michael Pleshakov <[email protected]>
Copy link
Contributor

@pleshakov pleshakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@LorcanMcVeigh LorcanMcVeigh merged commit 7139569 into master Aug 24, 2020
@LorcanMcVeigh LorcanMcVeigh deleted the reload-reason branch January 18, 2021 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants