Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics reported for deleted layers #172

Open
avacaru opened this issue Dec 10, 2020 · 0 comments
Open

Metrics reported for deleted layers #172

avacaru opened this issue Dec 10, 2020 · 0 comments

Comments

@avacaru
Copy link

avacaru commented Dec 10, 2020

Description
Metrics are produced for non-existent layers.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy a layer
  2. Check that metrics are produced correctly.
  3. Delete the layer
  4. Metrics (in particular reconcile_condition) are still produced for the deleted layer

Expected behavior
Once I delete a layer I expect no metrics to be produced from that point onwards.

Logs

root@node1:/# curl http://10.123.123.156:8080/metrics | grep reconcile_condition
# HELP reconcile_condition The current condition status of a GitOps Toolkit resource reconciliation.
# TYPE reconcile_condition gauge
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Deleted",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Deleted",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="False",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="True",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer1",status="Unknown",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Deleted",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Deleted",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="False",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="True",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer2",status="Unknown",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Deleted",type="Deployed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Deleted",type="Failed"} 1
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="False",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="False",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="True",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="True",type="Failed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Unknown",type="Deployed"} 0
reconcile_condition{kind="AddonsLayer.kraan.io",name="",namespace="layer3",status="Unknown",type="Failed"} 0

adi@WSL$ k get addonslayers.kraan.io
NAME                  VERSION             SOURCE                      PATH                            STATUS     MESSAGE
layer1                0.4.1               layer1-git                  ./add-ons/helm-releases         Deployed   AddonsLayer version 0.4.1 is Deployed, All HelmReleases deployed
layer2                1.0.2               layer2-git                  ./add-ons/helm-releases         Deployed   AddonsLayer version 1.0.2 is Deployed, All HelmReleases deployed

Additional context
As you can see above, layer 3 has been deleted (long time before querying the metrics endpoint) but the reconcile_condition metric is still produced for it. I'm trying to create an alert for reconcile_condition{type="Deployed", status="True"} < 0, but it keeps triggering for layers that don't exist anymore.

Another observation here would be that the name of the layer is in the namespace field instead of the name field. I'd appreciate if that could be fixed as well. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant