Skip to content

Add post upgrade check on capability #45010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-ci merged 1 commit into from
Nov 6, 2023
Merged

Add post upgrade check on capability #45010

openshift-ci merged 1 commit into from
Nov 6, 2023

Conversation

@shellyyang1989
Copy link
Contributor

@shellyyang1989 shellyyang1989 commented Oct 30, 2023

Refers OCPQE-17567

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 30, 2023

@shellyyang1989, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not determine changed registry steps: could not load step registry: test `cucushift-upgrade-check-capability` has `commands` containing `trap` command, but test step is missing grace_period
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci openshift-ci bot requested review from jhou1 and neisw October 30, 2023 03:11
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 30, 2023

@shellyyang1989, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not determine changed registry steps: could not load step registry: test `cucushift-upgrade-check-capability` has `commands` containing `trap` command, but test step is missing grace_period
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@shellyyang1989 shellyyang1989 force-pushed the check-cap-post-upgrade branch 3 times, most recently from d6a9f8f to fb93ae4 Compare October 30, 2023 03:36
@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Oct 30, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Oct 30, 2023

@jianlinliu @jinyunma @jiajliu @JianLi-RH Although it doesn't pass test yet, I need more eyes to make sure the logic makes sense. PTAL when you have time. Thank you!
cc @evakhoni

jianlinliu
jianlinliu reviewed Oct 30, 2023
View reviewed changes
...p-registry/cucushift/upgrade/check/capability/cucushift-upgrade-check-capability-commands.sh Outdated
Copy link
Contributor

@jianlinliu jianlinliu Oct 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we exit it as failure, if we are not very confident with that if the check is ready to move forward? Or should we always exit as passed, to check the log in early stage?

Copy link
Contributor Author

@shellyyang1989 shellyyang1989 Oct 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we exit 0, we will not get alert about the error so we won't know it failed until we check the log actively. Considering the test introduced critical bugs, I'm creating a Junit for it and thinking about to merge it to upgrade RP attribute. So we will be notified when the step fails

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Oct 30, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Oct 30, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28

jiajliu
jiajliu reviewed Oct 31, 2023
View reviewed changes
...p-registry/cucushift/upgrade/check/capability/cucushift-upgrade-check-capability-commands.sh Outdated
Copy link
Contributor

@jiajliu jiajliu Oct 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source _version should be the last upgraded version in chain upgrade?

Copy link
Contributor Author

@shellyyang1989 shellyyang1989 Oct 31, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a separate step which is out of chain upgrade step. So it's expected to be run once in chain upgrade. It validates the caps on the upgraded cluster compared to the initial cluster.

Copy link
Contributor

@jiajliu jiajliu Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do u mean, if it's 4.11-4.14 upgrade, it only do post check on cap for the last hop 4.14, while to compare with a 4.11 cluster? hmm, i wonder if the post check fail, is it easy for us to identify which hop got a wrong cap.

Copy link
Contributor Author

@shellyyang1989 shellyyang1989 Oct 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's the drawback. But I don't prefer to move it to upgrade step as the step is complicated enough and it will be more complex with more caps introduced. It's not cvo only feature and all cap related components are supposed to ensure the functionality. I'm hoping other qe teams to treat it carefully in their e2e automation.

Copy link
Contributor

@jiajliu jiajliu Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a proper job for a test against chain upgrade on such profile?

Copy link
Contributor Author

@shellyyang1989 shellyyang1989 Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have chain upgrade job on caps profile currently. If someday we decide to add such job, we can update the step as needed, but I'm afraid that would be complex

@shellyyang1989 shellyyang1989 force-pushed the check-cap-post-upgrade branch 2 times, most recently from 819663f to 19e7dd0 Compare October 31, 2023 06:16
@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Oct 31, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28

@shellyyang1989 shellyyang1989 force-pushed the check-cap-post-upgrade branch 3 times, most recently from 19f69a6 to 86ae1a4 Compare November 1, 2023 09:32
@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 1, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

1 similar comment
@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 1, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 1, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28

@jianlinliu
Copy link
Contributor

jianlinliu commented Nov 3, 2023

Looks good to me now.

Only one question, can we run some test job to check if we really can catch OCPBUGS-22718 by the PR? I think we can run pj-rehearse job on gcp-ipi-baselinecaps-none-additionalcaps-f7 to catch it, but seem like the latest nightly build already include the fix, maybe we can specify a lower target nightly build in this PR temporarily just for reproduce it? or we can add a stable -> stable upgrade job for gcp-ipi-baselinecaps-none-additionalcaps install workflow in this PR to run the pj-rehearse testing (I saw quay build does not include the fix yet)?

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 3, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

3 similar comments
@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 3, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 4, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 4, 2023

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 4, 2023

Rehearse job failed as expected.

actual_implicit_caps: Build Console DeploymentConfig ImageRegistry MachineAPI
 expected_implicit_caps: Build DeploymentConfig ImageRegistry MachineAPI
ERROR: Actual implicitly enabled capabilities don't match expected implicitly enabled capabilities
------check cvo status capabilities check-----
===check .status.capabilities.enabledCapabilities
ERROR: enabled capabilities does not match with cvo .status.capabilities.enabledCapabilities!
ERROR: capability Console: operator console -- found

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 4, 2023

@shellyyang1989: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28 ae9e4dbaf39ee8e59d84916b2706467c06a45f0b link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-baselinecaps-none-f28
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28 0841f5ae15c389f1dd14f20239155c95ff3deb3e link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-stable-4.14-upgrade-from-stable-4.13-aws-ipi-baselinecaps-none-f28

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Nov 5, 2023

[REHEARSALNOTIFIER]
@shellyyang1989: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-aws-ipi-imdsv2-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.10-gcp-ipi-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.12-azure-ipi-fullyprivate-proxy-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-vsphere-ipi-sdn-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-arm64-nightly-4.11-upgrade-from-stable-4.11-aws-ipi-proxy-sts-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-gcp-ipi-ovn-ipsec-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-aro-aro-f30 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-nightly-4.11-upgrade-from-stable-4.11-vsphere-ipi-ovn-ipsec-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-arm64-nightly-4.13-upgrade-from-stable-4.12-aws-ipi-proxy-sts-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-azure-ipi-disk-encryption-set-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-multi-nightly-4.15-upgrade-from-stable-4.14-azure-ipi-ovn-ipsec-amd-mixarch-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.14-gcp-upi-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.13-aws-ipi-ingress-lbtype-nlb-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.11-azure-ipi-fullyprivate-proxy-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-ibmcloud-ipi-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-ibmcloud-ipi-workers-rhel8-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-arm64-nightly-4.14-upgrade-from-stable-4.14-aws-ipi-ingress-lbtype-nlb-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.13-aws-ipi-byo-iam-role-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-arm64-nightly-4.15-upgrade-from-stable-4.14-azure-ipi-ovn-ipsec-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.11-vsphere-ipi-disk-encryption-tang-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.14-gcp-ipi-proxy-private-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.14-ibmcloud-ipi-private-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-vsphere-ipi-compact-etcd-encryption-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-nightly-4.11-upgrade-from-stable-4.10-azure-mag-ipi-fips-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.11-amd64-nightly-4.11-upgrade-from-stable-4.11-aws-usgov-ipi-private-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.10-amd64-nightly-4.10-upgrade-from-stable-4.9-aws-upi-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-arm64-nightly-4.13-upgrade-from-stable-4.13-aws-ipi-usertags-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.10-amd64-nightly-4.10-upgrade-from-stable-4.9-azure-ipi-ovn-ipsec-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-gcp-upi-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-4.15-upgrade-from-stable-4.14-aws-ipi-shared-vpc-phz-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.13-azure-ipi-ovn-ipsec-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-amd64-nightly-4.13-upgrade-from-stable-4.12-aws-usgov-ipi-private-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-arm64-nightly-4.15-upgrade-from-stable-4.14-azure-ipi-disk-encryption-set-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.13-ibmcloud-ipi-private-fips-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.12-amd64-nightly-4.12-upgrade-from-stable-4.12-azure-ipi-private-f28 N/A periodic Registry content changed

A total of 461 jobs have been affected by this change. The above listing is non-exhaustive and limited to 35 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 10 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 20 rehearsals
Comment: /pj-rehearse max to run up to 35 rehearsals
Comment: /pj-rehearse auto-ack to run up to 10 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@jianlinliu
Copy link
Contributor

jianlinliu commented Nov 6, 2023

/lgtm

@jianlinliu
Copy link
Contributor

jianlinliu commented Nov 6, 2023

/pj-rehearse ack

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Nov 6, 2023
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 6, 2023
@jianlinliu
Copy link
Contributor

jianlinliu commented Nov 6, 2023

cc @liangxia to review.

@liangxia
Copy link
Member

liangxia commented Nov 6, 2023

/approve

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 6, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianlinliu, liangxia, shellyyang1989

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 6, 2023
@openshift-ci openshift-ci bot merged commit e1a81db into openshift:master Nov 6, 2023
@jianlinliu
Copy link
Contributor

jianlinliu commented Nov 6, 2023

@shellyyang1989 shall we also apply the checking to openshift-upgrade-qe-test-disconnected chain?

@shellyyang1989
Copy link
Contributor Author

shellyyang1989 commented Nov 6, 2023

@shellyyang1989 shall we also apply the checking to openshift-upgrade-qe-test-disconnected chain?

Sure. Didn't realize we have cap test on disconnected env

@jianzhangbjz jianzhangbjz mentioned this pull request Nov 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jhou1 jhou1 Awaiting requested review from jhou1

@neisw neisw Awaiting requested review from neisw

3 more reviewers

@jianlinliu jianlinliu jianlinliu left review comments

@jiajliu jiajliu jiajliu left review comments

@jinyunma jinyunma jinyunma left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@jianlinliu jianlinliu

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@shellyyang1989 @openshift-ci-robot @jianlinliu @liangxia @jiajliu @jinyunma