Skip to content

Conversation

@patrickdillon
Copy link
Contributor

Replaces #3473 in order to be MCO specific.

Resolved issues that were plaguing #3473 by updating rhel scaleup template. #3620 is the equivalent of this PR for the installer repo and the master branch has been passing.

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 10, 2019
@patrickdillon
Copy link
Contributor Author

/cc @runcom @vrutkovs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to clarify, you made some tweaks to rhel scaleup here and also reordered the upgrade & scaleup in this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kikisdeliveryservice these files are generated by prowgen. I'm guessing it tries to keep them in alphabetical order, which unfortunately is making this harder to review. As best as I can tell there were no unintended changes to the upgrade job.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh thank you!! did not realize!

@kikisdeliveryservice
Copy link
Contributor

kikisdeliveryservice commented May 10, 2019

Is there a reason why in both *presubmits.yaml, the order of the upgrades and the scale ups were switched and they weren't just modified in place? (it's harder to see the changes that way, but if there's a reason it's ok)

These are apparently generated files, which I did not realize. 👍

Copy link
Contributor

@vrutkovs vrutkovs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 10, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: patrickdillon, vrutkovs
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: lorbuschris

If they are not already assigned, you can assign the PR to them by writing /assign @lorbuschris in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@patrickdillon
Copy link
Contributor Author

here we go
/retest

first run failed with [sig-cli] Kubectl client [k8s.io] Simple pod should contain last line of the log [Suite:openshift/conformance/parallel] [Suite:k8s]

@patrickdillon
Copy link
Contributor Author

/retest

1 similar comment
@patrickdillon
Copy link
Contributor Author

/retest

@runcom
Copy link
Member

runcom commented May 12, 2019

Hold on here, the scaleup job isn't working properly and of little help anyway:

ip-10-0-138-138.ec2.internal   Ready    worker   9m16s   v1.13.4+af45cda     10.0.138.138   <none>        CentOS Linux 7 (Core)                                      3.10.0-957.el7.x86_64   cri-o://1.12.10-2.rhaos4.0.git2c94bb7.el7
ip-10-0-147-119.ec2.internal   Ready    worker   9m7s    v1.13.4+af45cda     10.0.147.119   <none>        CentOS Linux 7 (Core)                                      3.10.0-957.el7.x86_64   cri-o://1.12.10-2.rhaos4.0.git2c94bb7.el7

The scaleup job brings in 2 CentOS 7 worker, QE tests with RHEL7.6. RHEL has CRI-O 1.13 which is the right one, CentOS 7 has CRI-O 1.12 which is completely wrong. We need to align on this or this job isn't reflecting reality in CI and what we test

@patrickdillon
Copy link
Contributor Author

patrickdillon commented May 12, 2019

Hold on here, the scaleup job isn't working properly and of little help anyway:

#3655 should have fixed the main issues with the scaleup job and we are starting to see some green (release-4.2 won't pass, but the other branches do). I am hoping if we turn this on as non-blocking we can start determining what are the issues specific to the scaleup tests and what are just e2e or installer flakes/issues.

ip-10-0-138-138.ec2.internal   Ready    worker   9m16s   v1.13.4+af45cda     10.0.138.138   <none>        CentOS Linux 7 (Core)                                      3.10.0-957.el7.x86_64   cri-o://1.12.10-2.rhaos4.0.git2c94bb7.el7
ip-10-0-147-119.ec2.internal   Ready    worker   9m7s    v1.13.4+af45cda     10.0.147.119   <none>        CentOS Linux 7 (Core)                                      3.10.0-957.el7.x86_64   cri-o://1.12.10-2.rhaos4.0.git2c94bb7.el7

The scaleup job brings in 2 CentOS 7 worker, QE tests with RHEL7.6. RHEL has CRI-O 1.13 which is the right one, CentOS 7 has CRI-O 1.12 which is completely wrong. We need to align on this or this job isn't reflecting reality in CI and what we test

@mtnbikenc @vrutkovs What do you think about this?

@runcom
Copy link
Member

runcom commented May 12, 2019

@mtnbikenc @vrutkovs What do you think about this?

the kubelet isn't also the one we're targeting with RHCOS (please check that as well)

@runcom
Copy link
Member

runcom commented May 12, 2019

also, by "scaleup is of little help" I didn't mean to say the job per-se isn't valuable. Actually, I can see that the scaleup works great 99.999% of the time, what doesn't work isn't the scaleup, it's probably the mix of wrong components which make openshift tests fail for some reason. My aim is indeed to have this job setup properly and working!

@vrutkovs
Copy link
Contributor

QE tests with RHEL7.6. RHEL has CRI-O 1.13 which is the right one, CentOS 7 has CRI-O 1.12 which is completely wrong. We need to align on this or this job isn't reflecting reality in CI and what we test

See #3759

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 14, 2019
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label May 16, 2019
@openshift-ci-robot
Copy link
Contributor

New changes are detected. LGTM label has been removed.

@openshift-ci-robot openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 16, 2019
@patrickdillon
Copy link
Contributor Author

/retest

3 similar comments
@patrickdillon
Copy link
Contributor Author

/retest

@patrickdillon
Copy link
Contributor Author

/retest

@patrickdillon
Copy link
Contributor Author

/retest

@patrickdillon
Copy link
Contributor Author

@runcom it looks like #3761 will take care of the crio version issue. I'm not certain about the kubelet and will check that. This PR keeps the tests optional, do you want to see these changes before merging this PR?

@patrickdillon
Copy link
Contributor Author

rebased

@patrickdillon
Copy link
Contributor Author

/retest

@openshift-ci-robot
Copy link
Contributor

@patrickdillon: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/rehearse/openshift/machine-config-operator/release-4.3/e2e-aws-scaleup-rhel7 681f67a link /test pj-rehearse
ci/rehearse/openshift/machine-config-operator/master/e2e-aws-scaleup-rhel7 681f67a link /test pj-rehearse
ci/rehearse/openshift/machine-config-operator/release-4.2/e2e-aws-scaleup-rhel7 681f67a link /test pj-rehearse
ci/prow/pj-rehearse 681f67a link /test pj-rehearse

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link
Contributor

@patrickdillon: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2019
@patrickdillon
Copy link
Contributor Author

Scaleup tests are broken and we are waiting for updates to crio repos. Will reopen once scaleup tests are fixed.

@mtnbikenc
Copy link
Member

@patrickdillon The tests should be fixed now. crio is updated and we create the necessary dirs required for pods to start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants