Bug 1752814: Fix decreasing counter metrics when reloading HAProxy #179

sgreene570 · 2020-09-11T14:11:01Z

The HAProxy router reload process calls CollectNow() in pkg/router/metrics/haproxy/haproxy.go. Currently, CollectNow() does not update e.lastScrape, which means it is possible for the normal scrape function Collect(...) to be called just after CollectNow(), but before HAProxy has actually reloaded, since CollectNow() essentially "resets" the metrics scrape interval.

The idea in this PR is to have CollectNow() set the e.lastScrape time, since CollectNow() is calling scrape() anyways. This will prevent counter metrics from decreasing across reloads since metrics are only saved for reload preservation during CollectNow() to minimize memory usage.

openshift-ci-robot · 2020-09-11T18:15:20Z

@sgreene570: This pull request references Bugzilla bug 1752814, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-09-11T18:18:19Z

@sgreene570: This pull request references Bugzilla bug 1752814, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sgreene570 · 2020-09-11T18:19:11Z

/assign @frobware @Miciah
/cc @danehans @knobunc

frobware · 2020-09-14T18:39:17Z

I would be interested to know whether this changes the behaviour of openshift/origin#25484

sgreene570 · 2020-09-14T18:43:29Z

I would be interested to know whether this changes the behaviour of openshift/origin#25484

Definitely could be the case.
Let me try to reproduce the test failure from openshift/origin#25484 and see if this PR remedies the situation at all.

sgreene570 · 2020-09-15T20:44:26Z

/test e2e

sgreene570 · 2020-09-15T22:13:55Z

/test e2e

sgreene570 · 2020-09-16T16:35:15Z

Re-triggering e2e test to get an idea if this PR hits https://bugzilla.redhat.com/show_bug.cgi?id=1835371

/test e2e

sgreene570 · 2020-09-16T17:09:31Z

/test e2e

sgreene570 · 2020-09-16T17:29:16Z

gcp quota problems. Will retest later today.

sgreene570 · 2020-09-16T19:20:29Z

/retest

sgreene570 · 2020-09-16T19:44:46Z

/retest

sgreene570 · 2020-09-16T20:46:22Z

/test e2e

sgreene570 · 2020-09-17T13:32:15Z

/test e2e

sgreene570 · 2020-09-17T17:42:44Z

/test e2e

sgreene570 · 2020-09-17T21:34:10Z

/retest

danehans · 2020-09-18T16:21:02Z

@sgreene570 should pkg/router/metrics/haproxy/haproxy_test.go be updated? It looks like the test is setting e.lastScrape which should be removed due to the changes introduced by this PR, right?

sgreene570 · 2020-09-18T16:27:45Z

@sgreene570 should pkg/router/metrics/haproxy/haproxy_test.go be updated? It looks like the test is setting e.lastScrape which should be removed due to the changes introduced by this PR, right?

Good point! Thanks for catching this.
Some of the time in haproxy_test.go, e.lastScrape is set intentionally to test that the scrape interval is working. That being said, I did find one instance of e.lastScrape = that can be removed from the test file in this PR. Thanks!

sgreene570 · 2020-09-18T16:33:50Z

go: github.com/openshift/[email protected] requires
	bitbucket.org/ww/[email protected]: reading https://api.bitbucket.org/2.0/repositories/ww/goautoneg?fields=scm: 404 Not Found

Looks like library-go needs to be bumped for the router as well. I'll amends this bump to #187

sgreene570 · 2020-09-18T16:58:15Z

Blocked on #187

sgreene570 · 2020-09-21T13:52:50Z

/test verify

danehans · 2020-09-21T15:58:26Z

/lgtm

openshift-ci-robot · 2020-09-21T15:58:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, sgreene570

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [sgreene570]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-09-21T17:07:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-21T17:46:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-09-21T18:58:31Z

@sgreene570: All pull requests linked via external trackers have merged:

openshift/router#179

Bugzilla bug 1752814 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sgreene570 · 2020-10-22T13:31:27Z

/cherry-pick release-4.5

openshift-cherrypick-robot · 2020-10-22T13:31:36Z

@sgreene570: new pull request created: #205

Details

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sgreene570 · 2020-10-22T13:31:50Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-10-22T13:31:57Z

@sgreene570: new pull request created: #206

Details

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 11, 2020

openshift-ci-robot requested review from frobware and knobunc September 11, 2020 14:11

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 11, 2020

sgreene570 changed the title ~~[WIP] Attempt to fix decreasing metrics counters when HAProxy reloads.~~ Bug 1752814: Fix decreasing counter metrics when reloading HAProxy Sep 11, 2020

openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Sep 11, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Sep 11, 2020

sgreene570 force-pushed the metrics-reload-drops branch from f3c774b to 8fb1ed4 Compare September 11, 2020 18:15

openshift-ci-robot assigned frobware and Miciah Sep 11, 2020

frobware mentioned this pull request Sep 15, 2020

[WIP] router/metrics: experiment with metrics refresh openshift/origin#25484

Closed

Fix decreasing metric counters when reloading HAProxy

29a33b1

sgreene570 force-pushed the metrics-reload-drops branch from 8fb1ed4 to 29a33b1 Compare September 18, 2020 16:28

openshift-ci-robot assigned danehans Sep 21, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 21, 2020

openshift-merge-robot merged commit 63622ca into openshift:master Sep 21, 2020

openshift-cherrypick-robot mentioned this pull request Oct 22, 2020

[release-4.5] Bug 1890534: Fix decreasing counter metrics when reloading HAProxy #205

Merged

openshift-cherrypick-robot mentioned this pull request Oct 22, 2020

[release-4.4] Bug 1890536: Fix decreasing counter metrics when reloading HAProxy #206

Merged

sgreene570 mentioned this pull request Oct 22, 2020

[release-3.11] Bug 1890545: Backport Fix decreasing metric counters when reloading HAProxy openshift/origin#25623

Merged

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy #179

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy #179

Uh oh!

Conversation

sgreene570 commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 11, 2020

Uh oh!

openshift-ci-robot commented Sep 11, 2020

Uh oh!

sgreene570 commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

frobware commented Sep 14, 2020

Uh oh!

sgreene570 commented Sep 14, 2020

Uh oh!

sgreene570 commented Sep 15, 2020

Uh oh!

sgreene570 commented Sep 15, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 16, 2020

Uh oh!

sgreene570 commented Sep 17, 2020

Uh oh!

sgreene570 commented Sep 17, 2020

Uh oh!

sgreene570 commented Sep 17, 2020

Uh oh!

danehans commented Sep 18, 2020

Uh oh!

sgreene570 commented Sep 18, 2020

Uh oh!

sgreene570 commented Sep 18, 2020

Uh oh!

sgreene570 commented Sep 18, 2020

Uh oh!

sgreene570 commented Sep 21, 2020

Uh oh!

danehans commented Sep 21, 2020

Uh oh!

openshift-ci-robot commented Sep 21, 2020

Uh oh!

openshift-bot commented Sep 21, 2020

Uh oh!

openshift-bot commented Sep 21, 2020

Uh oh!

openshift-ci-robot commented Sep 21, 2020

Uh oh!

sgreene570 commented Oct 22, 2020

Uh oh!

openshift-cherrypick-robot commented Oct 22, 2020

Uh oh!

sgreene570 commented Oct 22, 2020

Uh oh!

openshift-cherrypick-robot commented Oct 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

sgreene570 commented Sep 11, 2020 •

edited

Loading

sgreene570 commented Sep 11, 2020 •

edited

Loading