Skip to content

Conversation

@ardaguclu
Copy link
Member

@ardaguclu ardaguclu commented Jan 11, 2022

In disconnected upgrades, namely e2e-metal-ipi-upgrade-ovn-ipv6,
release image is mirrored to be used, however since ImageContentSourcePolicy
is not created, pods get image pull errors and upgrade fails in presubmit jobs.

Upgrade passes in periodics because from and to releases use same registry.

This PR adds ImageContentSourcePolicy.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 11, 2022
@ardaguclu ardaguclu changed the title Add to-release-image flag into disconnected mirror WIP: Add to-release-image flag into disconnected mirror Jan 11, 2022
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 11, 2022
@openshift-ci openshift-ci bot requested review from petr-muller and smg247 January 11, 2022 07:00
@ardaguclu ardaguclu force-pushed the mirror-ipv6-upgrade-to-release branch 5 times, most recently from 031aa17 to 4cfc9cc Compare January 11, 2022 11:12
@ardaguclu
Copy link
Member Author

/test ci/rehearse/openshift/metallb/release-4.10/e2e-metal-ipi-upgrade-ovn-ipv6

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 11, 2022

@ardaguclu: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test app-ci-config-dry
  • /test boskos-config
  • /test boskos-config-generation
  • /test build-clusters
  • /test build01-dry
  • /test build02-dry
  • /test build03-dry
  • /test build04-dry
  • /test ci-operator-config
  • /test ci-operator-config-metadata
  • /test ci-operator-registry
  • /test ci-secret-bootstrap-config-validation
  • /test ci-testgrid-allow-list
  • /test config
  • /test core-valid
  • /test correctly-sharded-config
  • /test deprecate-templates
  • /test generated-config
  • /test generated-dashboards
  • /test hive-dry
  • /test openshift-image-mirror-mappings
  • /test ordered-prow-config
  • /test owners
  • /test pj-rehearse-blocking
  • /test prow-config
  • /test prow-config-filenames
  • /test prow-config-semantics
  • /test pylint
  • /test release-config
  • /test release-controller-config
  • /test secret-generator-config-valid
  • /test services-valid
  • /test step-registry-metadata
  • /test step-registry-shellcheck
  • /test yamllint

The following commands are available to trigger optional jobs:

  • /test arm01-dry
  • /test pj-rehearse
  • /test pj-rehearse-max
  • /test pj-rehearse-more
  • /test vsphere-dry

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-release-ci-secret-bootstrap-config-validation
  • pull-ci-openshift-release-master-app-ci-config-dry
  • pull-ci-openshift-release-master-arm01-dry
  • pull-ci-openshift-release-master-boskos-config
  • pull-ci-openshift-release-master-boskos-config-generation
  • pull-ci-openshift-release-master-build-clusters
  • pull-ci-openshift-release-master-build01-dry
  • pull-ci-openshift-release-master-build02-dry
  • pull-ci-openshift-release-master-build03-dry
  • pull-ci-openshift-release-master-build04-dry
  • pull-ci-openshift-release-master-ci-operator-config
  • pull-ci-openshift-release-master-ci-operator-config-metadata
  • pull-ci-openshift-release-master-ci-operator-registry
  • pull-ci-openshift-release-master-config
  • pull-ci-openshift-release-master-core-valid
  • pull-ci-openshift-release-master-correctly-sharded-config
  • pull-ci-openshift-release-master-deprecate-templates
  • pull-ci-openshift-release-master-generated-config
  • pull-ci-openshift-release-master-generated-dashboards
  • pull-ci-openshift-release-master-hive-dry
  • pull-ci-openshift-release-master-ordered-prow-config
  • pull-ci-openshift-release-master-owners
  • pull-ci-openshift-release-master-pj-rehearse
  • pull-ci-openshift-release-master-prow-config
  • pull-ci-openshift-release-master-prow-config-filenames
  • pull-ci-openshift-release-master-prow-config-semantics
  • pull-ci-openshift-release-master-release-config
  • pull-ci-openshift-release-master-release-controller-config
  • pull-ci-openshift-release-master-secret-generator-config-valid
  • pull-ci-openshift-release-master-services-valid
  • pull-ci-openshift-release-master-step-registry-metadata
  • pull-ci-openshift-release-master-step-registry-shellcheck
  • pull-ci-openshift-release-master-vsphere-dry
  • pull-ci-openshift-release-openshift-image-mirror-mappings
  • pull-ci-openshift-release-yamllint
Details

In response to this:

/test ci/rehearse/openshift/metallb/release-4.10/e2e-metal-ipi-upgrade-ovn-ipv6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 11, 2022
@petr-muller
Copy link
Member

/uncc @smg247 @petr-muller

You'll need approvals from ci-operator/config/openshift/cluster-baremetal-operator/OWNERS

@ardaguclu ardaguclu force-pushed the mirror-ipv6-upgrade-to-release branch 5 times, most recently from e0820ca to 75ffa0b Compare January 13, 2022 06:51
@ardaguclu ardaguclu changed the title WIP: Add to-release-image flag into disconnected mirror WIP: Create ImageContentSourcePolicy for disconnected upgrade Jan 13, 2022
@ardaguclu ardaguclu force-pushed the mirror-ipv6-upgrade-to-release branch 2 times, most recently from 1702c65 to a0ae011 Compare January 13, 2022 06:58
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 13, 2022
@ardaguclu
Copy link
Member Author

@ardaguclu ardaguclu changed the title WIP: Create ImageContentSourcePolicy for disconnected upgrade Create ImageContentSourcePolicy for disconnected upgrade Jan 13, 2022
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 13, 2022
@andfasano
Copy link
Contributor

Changes look good to me, anyhow as discussed in another thread with @wking I think that moving also the upgrade release mirroring code in dev-scripts would be beneficial (to be done into another PR): all the mirroring code would stay in one place, and the task could be addressed during the setup step (and not during the test step, as it happens now).
Even more important, it will allow to easily test disconnected upgrades locally for devs.

@ardaguclu
Copy link
Member Author

Changes look good to me, anyhow as discussed in another thread with @wking I think that moving also the upgrade release mirroring code in dev-scripts would be beneficial (to be done into another PR): all the mirroring code would stay in one place, and the task could be addressed during the setup step (and not during the test step, as it happens now). Even more important, it will allow to easily test disconnected upgrades locally for devs.

Thanks @andfasano, it is a very good point and I agree with that. After that PR, we can move all mirroring in dev-scripts.

@ardaguclu
Copy link
Member Author

periodic-ci-openshift-release-master-nightly-4.10-e2e-metal-ipi-upgrade-ovn-ipv6 failed in setup stage which is not related to the code changes of this PR.

@ardaguclu
Copy link
Member Author

/assign @andfasano

could you please take a look?, if you'd like to see upgrade-ovn-ipv6 result, we can retest one more time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep this echo? I guess it would be useful when troubleshooting

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not overriding image anymore

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using actual image as upgrade and it is automatically mirrored to our registry via ImageContentSourcePolicy

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the canonical env var name used for injecting the release:latest pullspec and still used:

oc adm release mirror --registry-config ${DS_WORKING_DIR}/pull_secret.json \
 --from=${OPENSHIFT_UPGRADE_RELEASE_IMAGE_OVERRIDE} \

The echo will be just a tip in the log to specify to which version we're currently upgrading to

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But since we are not overriding, it is in;

openshift-tests run-upgrade all --to-image registry.build03.ci.openshift.org/ci-op-3g123y5y/release@sha256:5d19aec75540e8056cca0ab3ab1ee9c4098c0a703b1264ded57a477cae0c496e --provider '{"type":"baremetal","disconnected":true}' --from-repository virthost.ostest.test.metalkube.org:5000/localimages/local-test-image -o /logs/artifacts/e2e.log --junit-dir /logs/artifacts/junit

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_release/25130/rehearse-25130-pull-ci-openshift-cluster-baremetal-operator-master-e2e-metal-ipi-upgrade-ovn-ipv6-trigger/1481247658726133760/artifacts/e2e-metal-ipi-upgrade-ovn-ipv6-trigger/baremetalds-e2e-test/build-log.txt

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the var name it's a bit unfortunate, anyhow I noticed that the info I was really looking for is also reported in a better way later, ie:

"desired": {
      "version": "4.10.0-0.ci.test-2022-01-12-125752-ci-op-3g123y5y-latest",
      "image": "registry.build03.ci.openshift.org/ci-op-3g123y5y/release@sha256:5d19aec75540e8056cca0ab3ab1ee9c4098c0a703b1264ded57a477cae0c496e"
    }

@ardaguclu ardaguclu force-pushed the mirror-ipv6-upgrade-to-release branch from a0ae011 to 73b7dce Compare January 13, 2022 14:40
@openshift-ci openshift-ci bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 13, 2022
In disconnected upgrades, namely e2e-metal-ipi-upgrade-ovn-ipv6,
release image is mirrored to be used, however since ImageContentSourcePolicy
is not created, pods get image pull errors and upgrade fails in presubmit jobs.

Upgrade passes in periodics because from and to releases use same registry.

This PR adds ImageContentSourcePolicy.
@ardaguclu ardaguclu force-pushed the mirror-ipv6-upgrade-to-release branch from ce2d1b6 to 71b29a1 Compare January 14, 2022 06:06
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 14, 2022
@andfasano
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 14, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andfasano, ardaguclu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 14, 2022

@ardaguclu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/assisted-service/release-ocm-2.4/e2e-metal-assisted-operator-disconnected 4cfc9cc4d62e51ac6a9a71ca0886ce98977b3e5b link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-assisted-service-release-ocm-2.4-e2e-metal-assisted-operator-ztp-periodic-upgrade 4cfc9cc4d62e51ac6a9a71ca0886ce98977b3e5b link unknown /test pj-rehearse
ci/rehearse/openshift/cluster-baremetal-operator/master/e2e-metal-ipi-upgrade-ovn-ipv6 1702c657b8e614239dc071cbb6ff05d35a1a258a link unknown /test pj-rehearse
ci/rehearse/openshift/router/release-4.9/e2e-metal-ipi-ovn-router 71b29a1 link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.9-e2e-metal-ipi-ovn-dualstack 71b29a1 link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.10-e2e-metal-ipi-serial-virtualmedia ce2d1b638bf2f4851fc07899806cc1072840c2cb link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.10-e2e-metal-ipi-serial-ovn-dualstack 71b29a1 link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.10-e2e-metal-ipi-serial-compact 71b29a1 link unknown /test pj-rehearse
ci/prow/pj-rehearse 71b29a1 link false /test pj-rehearse

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 2845c10 into openshift:master Jan 14, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 14, 2022

@ardaguclu: Updated the step-registry configmap in namespace ci at cluster app.ci using the following files:

  • key baremetalds-e2e-test-commands.sh using file ci-operator/step-registry/baremetalds/e2e/test/baremetalds-e2e-test-commands.sh
Details

In response to this:

In disconnected upgrades, namely e2e-metal-ipi-upgrade-ovn-ipv6,
release image is mirrored to be used, however since ImageContentSourcePolicy
is not created, pods get image pull errors and upgrade fails in presubmit jobs.

Upgrade passes in periodics because from and to releases use same registry.

This PR adds ImageContentSourcePolicy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants