Use context to add timeout to cincinnati HTTP request #410

jottofar · 2020-07-17T17:20:40Z

Change CVO wait.Until instances to wait.UntilWithContext making top-level context available to the lower levels of the code. Use the added context to add a timeout to the remote cincinnati HTTP call which has the potential to hang the caller at the mercy of the target server.

The timeout has been set to an hour which was determined to be long enough to allow all expected well-behaved requests to complete but short enough to not be catastrophic if the calling thread were to hang.

wking · 2020-07-18T03:02:50Z

unit includes:

# github.com/openshift/cluster-version-operator/pkg/cvo [github.com/openshift/cluster-version-operator/pkg/cvo.test]
pkg/cvo/cvo_scenarios_test.go:114:15: not enough arguments in call to o.sync
	have (string)
	want (context.Context, string)

jottofar · 2020-07-21T12:44:03Z

/retest

jottofar · 2020-07-21T13:12:05Z

/test unit

jottofar · 2020-07-21T14:39:16Z

/retest

jottofar · 2020-07-22T15:24:39Z

/retest

jottofar · 2020-07-23T12:29:26Z

/retest

jottofar · 2020-07-23T17:04:00Z

/retest

pkg/cvo/sync_worker.go

pkg/cvo/cvo_test.go

pkg/cvo/cvo.go

pkg/cincinnati/cincinnati_test.go

jottofar · 2020-07-24T21:31:40Z

/retitle Use context to add timeout to cincinnati HTTP request

pkg/cincinnati/cincinnati.go

wking · 2020-07-30T13:35:32Z

pkg/cincinnati/cincinnati.go:89:36: undefined: transport

Change CVO wait.Until instances to wait.UntilWithContext making top-level context available to the lower levels of the code. Use the added context to add a timeout to the remote cincinnati HTTP call which has the potential to hang the caller at the mercy of the target server. The timeout has been set to an hour which was determined to be long enough to allow all expected well-behaved requests to complete but short enough to not be catastrophic if the calling thread were to hang.

jottofar · 2020-07-30T14:31:16Z

/test integration

jottofar · 2020-07-30T14:41:24Z

/test e2e

jottofar · 2020-07-30T19:45:35Z

/test e2e-upgrade

wking · 2020-07-31T01:50:29Z

/lgtm

openshift-ci-robot · 2020-07-31T01:50:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jottofar, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jottofar,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wking · 2020-07-31T03:53:58Z

e2e passed, except for an end user can use OLM can subscribe to the operator, which is rhbz#1862322.

/override ci/prow/e2e

openshift-ci-robot · 2020-07-31T03:54:15Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e

Details

In response to this:

e2e passed, except for an end user can use OLM can subscribe to the operator, which is rhbz#1862322.

/override ci/prow/e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Address a bug introduced by cc1921d (pkg/start: Release leader lease on graceful shutdown, 2020-08-03, openshift#424), where canceling the Operator.Run context would leave the operator with no time to attempt the final sync [1]: E0119 22:24:15.924216 1 cvo.go:344] unable to perform final sync: context canceled With this commit, I'm piping through shutdownContext, which gets a two-minute grace period beyond runContext, to give the operator time to push out that final status (which may include important information like the fact that the incoming release image has completed verification). --- This commit picks c4ddf03 (pkg/cvo: Use shutdownContext for final status synchronization, 2021-01-19, openshift#517) back to 4.5. It's not a clean pick, because we're missing changes like: * b72e843 (Bug 1822844: Block z level upgrades if ClusterVersionOverridesSet set, 2020-04-30, openshift#364). * 1d1de3b (Use context to add timeout to cincinnati HTTP request, 2019-01-15, openshift#410). which also touched these lines. But we've gotten this far without backporting rhbz#1822844, and openshift#410 was never associated with a bug in the first place, so instead of pulling back more of 4.6 to get a clean pick, I've just manually reconciled the pick conflicts. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1916384#c10

Address a bug introduced by cc1921d (pkg/start: Release leader lease on graceful shutdown, 2020-08-03, openshift#424), where canceling the Operator.Run context would leave the operator with no time to attempt the final sync [1]: E0119 22:24:15.924216 1 cvo.go:344] unable to perform final sync: context canceled With this commit, I'm piping through shutdownContext, which gets a two-minute grace period beyond runContext, to give the operator time to push out that final status (which may include important information like the fact that the incoming release image has completed verification). --- This commit picks c4ddf03 (pkg/cvo: Use shutdownContext for final status synchronization, 2021-01-19, openshift#517) back to 4.5. It's not a clean pick, because we're missing changes like: * b72e843 (Bug 1822844: Block z level upgrades if ClusterVersionOverridesSet set, 2020-04-30, openshift#364). * 1d1de3b (Use context to add timeout to cincinnati HTTP request, 2019-01-15, openshift#410). which also touched these lines. But we've gotten this far without backporting rhbz#1822844, and openshift#410 was never associated with a bug in the first place, so instead of pulling back more of 4.6 to get a clean pick, I've just manually reconciled the pick conflicts. Removing Start from pkg/start (again) fixes a buggy re-introduction in the manually-backported 20421b6 (*: Add lots of Context and options arguments, 2020-07-24, openshift#470). [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1916384#c10

openshift-ci-robot requested review from sdodson and smarterclayton July 17, 2020 17:20

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 17, 2020

jottofar force-pushed the ota-244-add-context branch 3 times, most recently from 3970ccf to 17b5248 Compare July 17, 2020 22:11

wking mentioned this pull request Jul 18, 2020

pkg/cincinnati: scope HTTP request to a context #83

Closed

jottofar force-pushed the ota-244-add-context branch 5 times, most recently from 4ffb363 to 9ebfffc Compare July 20, 2020 23:59

jottofar force-pushed the ota-244-add-context branch from 9ebfffc to 4923ce5 Compare July 21, 2020 12:49

wking reviewed Jul 23, 2020

View reviewed changes

pkg/cvo/sync_worker.go Outdated Show resolved Hide resolved

wking reviewed Jul 23, 2020

View reviewed changes

pkg/cvo/cvo_test.go Outdated Show resolved Hide resolved

wking reviewed Jul 23, 2020

View reviewed changes

pkg/cvo/cvo.go Outdated Show resolved Hide resolved

wking reviewed Jul 23, 2020

View reviewed changes

pkg/cincinnati/cincinnati_test.go Outdated Show resolved Hide resolved

jottofar force-pushed the ota-244-add-context branch from 4923ce5 to f5654a9 Compare July 24, 2020 21:30

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2020

openshift-ci-robot changed the title ~~Add a context to remote cincinnati calls from CVO~~ Use context to add timeout to cincinnati HTTP request Jul 24, 2020

jottofar force-pushed the ota-244-add-context branch from f5654a9 to 292f41e Compare July 24, 2020 21:40

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2020

wking mentioned this pull request Jul 24, 2020

Bump client-go and library-go to current 4.6 tips #406

Merged

wking reviewed Jul 27, 2020

View reviewed changes

pkg/cincinnati/cincinnati.go Outdated Show resolved Hide resolved

wking reviewed Jul 27, 2020

View reviewed changes

pkg/cincinnati/cincinnati.go Outdated Show resolved Hide resolved

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2020

jottofar force-pushed the ota-244-add-context branch from d5b7ee7 to d1af4f9 Compare July 28, 2020 18:24

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2020

jottofar force-pushed the ota-244-add-context branch 3 times, most recently from c67db6e to 8ee0d52 Compare July 30, 2020 13:20

jottofar force-pushed the ota-244-add-context branch from 8ee0d52 to 6cb435f Compare July 30, 2020 13:45

jottofar force-pushed the ota-244-add-context branch from 6cb435f to 1d1de3b Compare July 30, 2020 14:10

openshift-ci-robot assigned wking Jul 31, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2020

openshift-merge-robot merged commit 4c3024d into openshift:master Jul 31, 2020

wking mentioned this pull request Jul 31, 2020

vendor: Bump client-go and library-go to current 4.6 tips (2020-07-24) #420

Merged

This was referenced Aug 27, 2020

Bug 1872906: pkg/start: Release leader lease on graceful shutdown #445

Closed

Bug 1872906: pkg/start: Release leader lease on graceful shutdown #446

Merged

wking mentioned this pull request Feb 20, 2021

Bug 1931025: pkg/cvo: Use shutdownContext for final status synchronization #525

Merged

Use context to add timeout to cincinnati HTTP request #410

Use context to add timeout to cincinnati HTTP request #410

Uh oh!

Conversation

jottofar commented Jul 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Jul 18, 2020

Uh oh!

jottofar commented Jul 21, 2020

Uh oh!

jottofar commented Jul 21, 2020

Uh oh!

jottofar commented Jul 21, 2020

Uh oh!

jottofar commented Jul 22, 2020

Uh oh!

jottofar commented Jul 23, 2020

Uh oh!

jottofar commented Jul 23, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jottofar commented Jul 24, 2020

Uh oh!

Uh oh!

Uh oh!

wking commented Jul 30, 2020

Uh oh!

jottofar commented Jul 30, 2020

Uh oh!

jottofar commented Jul 30, 2020

Uh oh!

jottofar commented Jul 30, 2020

Uh oh!

wking commented Jul 31, 2020

Uh oh!

openshift-ci-robot commented Jul 31, 2020

Uh oh!

wking commented Jul 31, 2020

Uh oh!

openshift-ci-robot commented Jul 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jottofar commented Jul 17, 2020 •

edited

Loading