Skip to content

Conversation

@wking
Copy link
Member

@wking wking commented Nov 19, 2020

The outgoing integration tests have used a master-sidecar since they landed in d1af815 (#888). This recently broke, with jobs like this one failing with:

Waiting for kube-apiserver to come up
!!! [1118 23:29:05] Timed out waiting for kube-apiserver:  to answer at https://10.130.11.147:8443/healthz; tried 60 waiting 1 between each

With the Kubernetes API-server dying with:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/opensift_cluster-version-operator/443/pull-ci-openshift-cluster-version-operator-master-integration/1329201271097266176/artifacts/integration/logs/kube-apiserver.log | tail -n2
I1118 23:28:05.064590     308 flags.go:59] FLAG: --watch-cache-sizes="[]"
Error: error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied

This commit pivots to the off-the-shelf openshift-e2e-gcp workflow, substituting in a custom test step to run the existing integration tests.

WIP, because the amd64-only prowgen image:

$ oc image info -o json registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest | jq -r .configarchitecture
amd64

is not compatible with my current arm64 box. We'll see if CI tells me what make update would have done.

@openshift-ci-robot openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 19, 2020
@wking wking force-pushed the cvo-integration-via-steps branch from 0b92faf to c14cb55 Compare November 19, 2020 05:25
@wking
Copy link
Member Author

wking commented Nov 19, 2020

Patched as recommended by CI with 0b92faf656 -> c14cb5507f. We'll see how rehearsals go, and if this works I'll port back to the older release branches.

@jottofar
Copy link
Contributor

/retest

…rator tests

The outgoing integration tests have used a master-sidecar since they
landed in d1af815 (Integration tests for registry, replacing
existing, 2018-05-23, openshift#888).  This recently broke, with jobs like [1]
failing with:

  Waiting for kube-apiserver to come up
  !!! [1118 23:29:05] Timed out waiting for kube-apiserver:  to answer at https://10.130.11.147:8443/healthz; tried 60 waiting 1 between each

With the Kubernetes API-server dying with:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/opensift_cluster-version-operator/443/pull-ci-openshift-cluster-version-operator-master-integration/1329201271097266176/artifacts/integration/logs/kube-apiserver.log | tail -n2
  I1118 23:28:05.064590     308 flags.go:59] FLAG: --watch-cache-sizes="[]"
  Error: error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied

This commit pivots to the off-the-shelf openshift-e2e-gcp workflow,
substituting in a custom 'test' step to run the existing integration
tests.

Porting back to 4.4.  Older releases are end-of-life [2], but if we
need to revive them for some reason we can port this back further
then.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/443/pull-ci-openshift-cluster-version-operator-master-integration/1329201271097266176
[2]: https://access.redhat.com/support/policy/updates/openshift#dates
@wking wking force-pushed the cvo-integration-via-steps branch from bbbf63a to 5bd9761 Compare December 9, 2020 00:43
@wking wking changed the title WIP: ci-operator/config/openshift/cluster-version-operator: Multi-step operator tests ci-operator/config/openshift/cluster-version-operator: Multi-step operator tests Dec 9, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 9, 2020
@wking
Copy link
Member Author

wking commented Dec 9, 2020

Ok, I've pushed 4235b31 -> 5bd9761, expanding this to 4.4 and later, so I can drop the WIP. Earlier master presubmit looked good; we'll see how these go...

@wking
Copy link
Member Author

wking commented Dec 9, 2020

All green :).

/assign @jottofar

@LalatenduMohanty
Copy link
Member

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 9, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit def5d7a into openshift:master Dec 9, 2020
@openshift-ci-robot
Copy link
Contributor

@wking: Updated the following 18 configmaps:

  • job-config-4.4 configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-release-4.4-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.4-presubmits.yaml
  • job-config-4.6 configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.6-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.6-presubmits.yaml
  • job-config-4.7 configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.7-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.7-presubmits.yaml
  • job-config-4.8 configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.8-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.8-presubmits.yaml
  • ci-operator-4.6-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.6.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.6.yaml
  • job-config-master configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-master-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-master-presubmits.yaml
  • job-config-4.5 configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.5-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.5-presubmits.yaml
  • job-config-4.6 configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-release-4.6-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.6-presubmits.yaml
  • job-config-4.8 configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-release-4.8-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.8-presubmits.yaml
  • ci-operator-4.7-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.7.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.7.yaml
  • ci-operator-4.8-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.8.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.8.yaml
  • job-config-4.5 configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-release-4.5-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.5-presubmits.yaml
  • job-config-4.7 configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-release-4.7-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.7-presubmits.yaml
  • ci-operator-4.5-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.5.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.5.yaml
  • job-config-master configmap in namespace ci at cluster api.ci using the following files:
    • key openshift-cluster-version-operator-master-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-master-presubmits.yaml
  • job-config-4.4 configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.4-presubmits.yaml using file ci-operator/jobs/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.4-presubmits.yaml
  • ci-operator-master-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-master.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-master.yaml
  • ci-operator-4.4-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-cluster-version-operator-release-4.4.yaml using file ci-operator/config/openshift/cluster-version-operator/openshift-cluster-version-operator-release-4.4.yaml
Details

In response to this:

The outgoing integration tests have used a master-sidecar since they landed in d1af815 (#888). This recently broke, with jobs like this one failing with:

Waiting for kube-apiserver to come up
!!! [1118 23:29:05] Timed out waiting for kube-apiserver: to answer at https://10.130.11.147:8443/healthz; tried 60 waiting 1 between each

With the Kubernetes API-server dying with:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/opensift_cluster-version-operator/443/pull-ci-openshift-cluster-version-operator-master-integration/1329201271097266176/artifacts/integration/logs/kube-apiserver.log | tail -n2
I1118 23:28:05.064590     308 flags.go:59] FLAG: --watch-cache-sizes="[]"
Error: error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied

This commit pivots to the off-the-shelf openshift-e2e-gcp workflow, substituting in a custom test step to run the existing integration tests.

WIP, because the amd64-only prowgen image:

$ oc image info -o json registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest | jq -r .configarchitecture
amd64

is not compatible with my current arm64 box. We'll see if CI tells me what make update would have done.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking deleted the cvo-integration-via-steps branch December 9, 2020 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants