MCO-654: forcefile should always trigger an OS update #3790

cdoern · 2023-07-11T14:10:56Z

- What I did

currently, the forcefile sometimes gets stopped from triggering an osupdate if one of our other priorities gets in the way. The forcefile should alwyas trigger and end to end update, re-applying all changes if the osImageURL is different. This will get many customers out of this half baked reboot loop we have been seeing.

- How to verify it

when using the forcefile, should always see Updating OS indicating a full reboot, especially when there is a config mismatch with the osImageURLs.

- Description for the changelog

the forcefile now always triggers a full os update.

openshift-ci-robot · 2023-07-11T14:11:20Z

@cdoern: This pull request references MCO-654 which is a valid jira issue.

Details

In response to this:

- What I did

currently, the forcefile sometimes gets stopped from triggering an osupdate if one of our other priorities gets in the way. The forcefile should alwyas trigger and en dot end update, re-applying all changes even is the osImageUrl is the same. This will get many customers out of this half baked reboot loop we have been seeing.

- How to verify it

when using the forcefile, should always see Updating OS indicating a full reboot.

- Description for the changelog

the forcefile now always triggers a full os update.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-07-11T14:11:30Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

cdoern · 2023-07-17T20:03:06Z

/retest-required

cdoern · 2023-07-18T14:19:12Z

/retest-required

djoshy · 2023-07-20T15:25:31Z

Looks like a forcefile test failure, might need some refactor since we are changing the behavior

djoshy · 2023-07-20T15:27:36Z

Also, this might need a rebase once #3583 lands

pkg/daemon/update.go

cdoern · 2023-08-07T13:01:13Z

Testing this with the suite is to costly. I ran tests by hand.

created a cluster with this PR.
found a new osImageURL from a nightly payload (rpm-ostree is the name)
ran rpm-ostree rebase --experimental ostree-unverified-registry:
waited for the daemon to mark the node as degraded, I applied the forcefile
node rebooted, marked as done with new osImageURL

openshift-ci-robot · 2023-08-07T13:15:10Z

@cdoern: This pull request references MCO-654 which is a valid jira issue.

Details

In response to this:

- What I did

currently, the forcefile sometimes gets stopped from triggering an osupdate if one of our other priorities gets in the way. The forcefile should alwyas trigger and en dot end update, re-applying all changes if the osImageURL is different. This will get many customers out of this half baked reboot loop we have been seeing.

- How to verify it

when using the forcefile, should always see Updating OS indicating a full reboot, especially when there is a config mismatch with the osImageURLs.

- Description for the changelog

the forcefile now always triggers a full os update.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-08-07T13:15:29Z

@cdoern: This pull request references MCO-654 which is a valid jira issue.

Details

In response to this:

- What I did

currently, the forcefile sometimes gets stopped from triggering an osupdate if one of our other priorities gets in the way. The forcefile should alwyas trigger and end to end update, re-applying all changes if the osImageURL is different. This will get many customers out of this half baked reboot loop we have been seeing.

- How to verify it

when using the forcefile, should always see Updating OS indicating a full reboot, especially when there is a config mismatch with the osImageURLs.

- Description for the changelog

the forcefile now always triggers a full os update.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/daemon/update.go

test/e2e-shared-tests/mcd_config_drift.go

pkg/daemon/update.go

sinnykumari · 2023-08-07T13:50:21Z

Also, we may want to get https://docs.openshift.com/container-platform/4.13/post_installation_configuration/machine-configuration-tasks.html updated for force file related change.

It will be good to add some bugs that we encountered around it and add it either to this PR or story.

Also, do you think this should go through qe testing?

cdoern · 2023-08-08T19:01:09Z

I do not think this needs QE testing as it is nothigh risk for any current functionality. @yuqi-zhang wdyt?

currently, the forcefile sometimes gets stopped from triggering an osupdate if one of our other priorities gets in the way. The forcefile should alwyas trigger and en dot end update, re-applying all changes even is the osImageUrl is the same. This will get many customers out of this half baked reboot loop we have been seeing. Signed-off-by: Charlie Doern <[email protected]>

sinnykumari · 2023-08-09T11:00:15Z

/lgtm
/retest

openshift-ci · 2023-08-09T11:05:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cdoern, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cdoern,sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-08-09T12:04:46Z

/retest-required

Remaining retests: 0 against base HEAD 40571ad and 2 for PR HEAD dc5f796 in total

cdoern · 2023-08-09T13:32:06Z

/retest-required

LorbusChris · 2023-08-09T21:39:39Z

/test okd-scos-images

LorbusChris · 2023-08-10T05:45:20Z

/test okd-scos-images

LorbusChris · 2023-08-10T06:02:11Z

/test okd-scos-e2e-aws-ovn

cdoern · 2023-08-10T12:03:17Z

/test e2e-hypershift

cdoern · 2023-08-10T18:28:15Z

/test e2e/hypershift

openshift-ci · 2023-08-10T18:31:40Z

@cdoern: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test 4.12-upgrade-from-stable-4.11-images
/test cluster-bootimages
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test images
/test okd-scos-images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade
/test bootstrap-unit
/test e2e-alibabacloud-ovn
/test e2e-aws-disruptive
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-fips-op
/test e2e-aws-ovn-workers-rhel8
/test e2e-aws-proxy
/test e2e-aws-serial
/test e2e-aws-single-node
/test e2e-aws-upgrade-single-node
/test e2e-aws-workers-rhel8
/test e2e-azure
/test e2e-azure-ovn-upgrade
/test e2e-azure-upgrade
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-rt
/test e2e-gcp-rt-op
/test e2e-gcp-single-node
/test e2e-gcp-upgrade
/test e2e-metal-assisted
/test e2e-metal-ipi
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6
/test e2e-openstack
/test e2e-openstack-externallb
/test e2e-openstack-parallel
/test e2e-ovirt
/test e2e-ovirt-upgrade
/test e2e-ovn-step-registry
/test e2e-vsphere
/test e2e-vsphere-upgrade
/test e2e-vsphere-upi
/test e2e-vsphere-upi-zones
/test e2e-vsphere-zones
/test okd-e2e-aws
/test okd-e2e-gcp-op
/test okd-e2e-upgrade
/test okd-e2e-vsphere
/test okd-images
/test okd-scos-e2e-aws-ovn
/test okd-scos-e2e-gcp-op
/test okd-scos-e2e-gcp-ovn-upgrade
/test okd-scos-e2e-vsphere

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-machine-config-operator-master-bootstrap-unit
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn
pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn-upgrade
pull-ci-openshift-machine-config-operator-master-e2e-gcp-op
pull-ci-openshift-machine-config-operator-master-e2e-gcp-op-single-node
pull-ci-openshift-machine-config-operator-master-e2e-hypershift
pull-ci-openshift-machine-config-operator-master-images
pull-ci-openshift-machine-config-operator-master-okd-images
pull-ci-openshift-machine-config-operator-master-okd-scos-e2e-aws-ovn
pull-ci-openshift-machine-config-operator-master-okd-scos-images
pull-ci-openshift-machine-config-operator-master-unit
pull-ci-openshift-machine-config-operator-master-verify

Details

In response to this:

/test e2e/hypershift

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cdoern · 2023-08-10T18:34:58Z

/test e2e-hypershift

openshift-ci · 2023-08-10T23:24:33Z

@cdoern: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci · 2023-08-11T15:54:12Z

/test remaining-required

cdoern changed the title ~~forcefile should always trigger an OS update~~ MCO-654: forcefile should always trigger an OS update Jul 11, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 11, 2023

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 11, 2023

cdoern marked this pull request as ready for review July 17, 2023 13:08

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2023

openshift-ci bot requested review from dkhater-redhat and jkyros July 17, 2023 13:12

cdoern force-pushed the forcefile branch from 96ee38a to 3bdf336 Compare July 18, 2023 15:03

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 20, 2023

cdoern force-pushed the forcefile branch from 3bdf336 to 5316659 Compare July 24, 2023 13:13

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2023

yuqi-zhang reviewed Jul 26, 2023

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

cdoern force-pushed the forcefile branch 2 times, most recently from 89c34ed to 436e327 Compare July 27, 2023 13:50

yuqi-zhang reviewed Aug 3, 2023

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

pkg/daemon/update.go Outdated Show resolved Hide resolved

cdoern force-pushed the forcefile branch 4 times, most recently from 13a0381 to d5bcf99 Compare August 4, 2023 19:45

sinnykumari reviewed Aug 7, 2023

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

sinnykumari reviewed Aug 7, 2023

View reviewed changes

test/e2e-shared-tests/mcd_config_drift.go Outdated Show resolved Hide resolved

sinnykumari reviewed Aug 7, 2023

View reviewed changes

pkg/daemon/update.go Outdated Show resolved Hide resolved

cdoern force-pushed the forcefile branch from d5bcf99 to dc5f796 Compare August 8, 2023 19:03

openshift-ci bot assigned sinnykumari Aug 9, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 9, 2023

openshift-merge-robot merged commit beca7a6 into openshift:master Aug 10, 2023

MCO-654: forcefile should always trigger an OS update #3790

MCO-654: forcefile should always trigger an OS update #3790

Uh oh!

Conversation

cdoern commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 11, 2023 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Jul 11, 2023

Uh oh!

cdoern commented Jul 17, 2023

Uh oh!

cdoern commented Jul 18, 2023

Uh oh!

djoshy commented Jul 20, 2023

Uh oh!

djoshy commented Jul 20, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cdoern commented Aug 7, 2023

Uh oh!

openshift-ci-robot commented Aug 7, 2023 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 7, 2023 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sinnykumari commented Aug 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdoern commented Aug 8, 2023

Uh oh!

sinnykumari commented Aug 9, 2023

Uh oh!

openshift-ci bot commented Aug 9, 2023

Uh oh!

openshift-ci-robot commented Aug 9, 2023

Uh oh!

cdoern commented Aug 9, 2023

Uh oh!

LorbusChris commented Aug 9, 2023

Uh oh!

LorbusChris commented Aug 10, 2023

Uh oh!

LorbusChris commented Aug 10, 2023

Uh oh!

cdoern commented Aug 10, 2023

Uh oh!

cdoern commented Aug 10, 2023

Uh oh!

openshift-ci bot commented Aug 10, 2023

Uh oh!

cdoern commented Aug 10, 2023

Uh oh!

openshift-ci bot commented Aug 10, 2023

Uh oh!

openshift-ci bot commented Aug 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cdoern commented Jul 11, 2023 •

edited

Loading

openshift-ci-robot commented Jul 11, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 7, 2023 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 7, 2023 •

edited by openshift-ci bot

Loading

sinnykumari commented Aug 7, 2023 •

edited

Loading