Skip to content

Bug 1763822: A timeout in applying an upgrade should not result in the CVO thinking that it is reconciling the payload #262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-merge-robot merged 1 commit into from
Oct 23, 2019
Merged

Bug 1763822: A timeout in applying an upgrade should not result in the CVO thinking that it is reconciling the payload #262

openshift-merge-robot merged 1 commit into from
Oct 23, 2019

Conversation

@openshift-cherrypick-robot
Copy link

@openshift-cherrypick-robot openshift-cherrypick-robot commented Oct 22, 2019

This is an automated cherry-pick of #255

/assign wking

@wking
pkg/payload/task_graph: RunGraph error when canceled
d0f801b 
The RunGraph implementation was unchanged since it landed in
cb4e037 (payload: Create a task graph that can split a payload into
chunks, 2019-01-17, openshift#88), with the exception of later logging and
c2ac20f (status: Report the operators that have not yet deployed,
2019-04-09, openshift#158) with the adjusted return type.

The old code launched a goroutine for the pushing/reaping, which was
an unecessary, and made error reporting on any outstanding tasks more
complicated.  I'd like to drop the goroutine, but Clayton is not
comfortable with backporting that large a change to older releases
[1].  And I'd like to be able to return errors like:

  1 incomplete task nodes, beginning with b

but Clayton thinks these are just "took too long, but we're still
making progress" and that they'll resolve on their own in the next
attempt or few, and that they're not actual deadlocks where you'd want
a better fingerprint to pin down the node(s) that were locking [2].

This commit ensures that when we are canceled we return an error, and
it does none of the refactoring we'd need to be able to say whether we
had unprocessed nodes (for late cancels, it's possible that we could
return "I was canceled" even if we had successfully pushed and reaped
all the nodes).  This should avoid situations like [3]:

  2019-10-21T10:34:30.63940461Z I1021 10:34:30.639073       1 start.go:19] ClusterVersionOperator v1.0.0-106-g0725bd53-dirty
  ...
  2019-10-21T10:34:31.132673574Z I1021 10:34:31.132635       1 sync_worker.go:453] Running sync quay.io/runcom/origin-release:v4.2-1196 (force=true) on generation 2 in state Updating at attempt 0
  ...
  2019-10-21T10:40:16.168632703Z I1021 10:40:16.168604       1 sync_worker.go:579] Running sync for customresourcedefinition "baremetalhosts.metal3.io" (101 of 432)
  2019-10-21T10:40:16.18425522Z I1021 10:40:16.184220       1 task_graph.go:583] Canceled worker 0
  2019-10-21T10:40:16.184381244Z I1021 10:40:16.184360       1 task_graph.go:583] Canceled worker 3
  ...
  2019-10-21T10:40:16.21772875Z I1021 10:40:16.217715       1 task_graph.go:603] Workers finished
  2019-10-21T10:40:16.217777479Z I1021 10:40:16.217759       1 task_graph.go:611] Result of work: []
  2019-10-21T10:40:16.217864206Z I1021 10:40:16.217846       1 task_graph.go:539] Stopped graph walker due to cancel
  ...
  2019-10-21T10:43:08.743798997Z I1021 10:43:08.743740       1 sync_worker.go:453] Running sync quay.io/runcom/origin-release:v4.2-1196 (force=true) on generation 2 in state Reconciling at attempt 0
  ...

where the CVO canceled some workers, saw that there are worker no
errors, and decided "upgrade complete" despite never having attempted
to push the bulk of its manifests.

Without the task_graph.go changes in this commit, the new test fails
with:

  $ go test -run TestRunGraph  ./pkg/payload/
  --- FAIL: TestRunGraph (1.03s)
      --- FAIL: TestRunGraph/cancelation_without_task_errors_is_reported (1.00s)
          task_graph_test.go:910: unexpected error: []
  FAIL
  FAIL		github.com/openshift/cluster-version-operator/pkg/payload				1.042s

Also change "cancelled" -> "canceled" to match Go's docs [4] and name
the other test cases.

[1]: openshift#255 (comment)
[2]: openshift#260
[3]: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1/754/artifacts/e2e-aws-upgrade/must-gather/registry-svc-ci-openshift-org-origin-4-1-sha256-f8c863ea08d64eea7b3a9ffbbde9c01ca90501afe6c0707e9c35f0ed7e92a9df/namespaces/openshift-cluster-version/pods/cluster-version-operator-5f5d465967-t57b2/cluster-version-operator/cluster-version-operator/logs/current.log
[4]: https://golang.org/pkg/context/#pkg-overview
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Oct 22, 2019
Merged
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 22, 2019

@openshift-cherrypick-robot: This pull request references Bugzilla bug 1763821, which is invalid:

  • expected the bug to target the "4.2.z" release, but it targets "4.3.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is MODIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[release-4.2] Bug 1763821: pkg/payload/task_graph: Canceling the task graph partway though is an error even if no tasks fail

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 22, 2019
@openshift-ci-robot openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 22, 2019
@wking wking changed the title [release-4.2] Bug 1763821: pkg/payload/task_graph: Canceling the task graph partway though is an error even if no tasks fail Bug 1763822: pkg/payload/task_graph: Canceling the task graph partway though is an error even if no tasks fail Oct 22, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 22, 2019

@openshift-cherrypick-robot: This pull request references Bugzilla bug 1763822, which is invalid:

  • expected dependent Bugzilla bug 1763821 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is MODIFIED instead
  • expected dependent Bugzilla bug 1763823 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is NEW instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1763822: pkg/payload/task_graph: Canceling the task graph partway though is an error even if no tasks fail

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@smarterclayton
Copy link
Contributor

smarterclayton commented Oct 22, 2019

/bugzilla refresh

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 22, 2019

@smarterclayton: This pull request references Bugzilla bug 1763822, which is invalid:

  • expected dependent Bugzilla bug 1763821 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is MODIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@smarterclayton
Copy link
Contributor

smarterclayton commented Oct 22, 2019

/lgtm
/approve

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 22, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 22, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2019
@smarterclayton smarterclayton changed the title Bug 1763822: pkg/payload/task_graph: Canceling the task graph partway though is an error even if no tasks fail Bug 1763822: A timeout in applying an upgrade should not result in the CVO thinking that it is reconciling the payload Oct 22, 2019
@jwforres jwforres added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Oct 22, 2019
@jwforres
Copy link
Member

jwforres commented Oct 22, 2019

Overriding bugzilla bot given urgency and manual testing that was done.

@jwforres jwforres added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Oct 22, 2019
@jwforres
Copy link
Member

jwforres commented Oct 22, 2019

/retest

@jwforres
Copy link
Member

jwforres commented Oct 22, 2019

failures both looked like AWS infra during install

@jwforres
Copy link
Member

jwforres commented Oct 22, 2019

/test e2e-aws-upgrade

@crawford
Copy link
Contributor

crawford commented Oct 22, 2019

listing hosted zones: Throttling: Rate exceeded

@jwforres
Copy link
Member

jwforres commented Oct 22, 2019

/test e2e-aws

@jwforres
Copy link
Member

jwforres commented Oct 23, 2019

starting to see successes in CI again, kicking it
/retest

@openshift-merge-robot openshift-merge-robot merged commit e5e4689 into openshift:release-4.2 Oct 23, 2019
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 23, 2019

@openshift-cherrypick-robot: All pull requests linked via external trackers have merged. Bugzilla bug 1763822 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1763822: A timeout in applying an upgrade should not result in the CVO thinking that it is reconciling the payload

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking mentioned this pull request Oct 29, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abhinavdahiya abhinavdahiya Awaiting requested review from abhinavdahiya

@crawford crawford Awaiting requested review from crawford

Assignees

@wking wking

@smarterclayton smarterclayton

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@openshift-cherrypick-robot @openshift-ci-robot @smarterclayton @jwforres @crawford @wking @openshift-merge-robot