openshift-e2e-test: Allow if no TEST_SKIPS match

[ardaguclu]

This PR prints warning message when tests not found in TEST_SKIPS and
allows CI steps continue.

There is a high probability that tests are not in TEST_SKIPS especially
for different TEST_SUITES(i.e. openshift/conformance/serial) and this
does not an indication of a failure.
Instead failing with an error message, process continues with an
warning message.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ardaguclu
To complete the pull request process, please assign wking after the PR has been reviewed.
You can assign the PR to them by writing /assign @wking in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/step-registry/openshift/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ardaguclu]

/cc @pierreprinetti @vrutkovs

[openshift-ci]

@ardaguclu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/origin/release-4.5/e2e-aws-csi	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/origin/release-4.1/e2e-aws-builds	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.9/e2e-gcp-upi-xpn	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/release-4.9/e2e-aws-manual-oidc	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-kube-controller-manager-operator/release-4.10/e2e-aws-ccm	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.10/e2e-gcp-ccm	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/aws-efs-csi-driver-operator/release-4.9/operator-e2e	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.10/e2e-gcp-ccm-install	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-logging-operator/tech-preview/e2e-operator	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/origin/release-4.2/e2e-cmd	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/machine-config-operator/release-4.9/e2e-gcp-single-node	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.9/e2e-kubevirt	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/kubernetes/release-4.9/e2e-openstack-csi-manila	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.9/e2e-gcp-shared-vpc	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.10/e2e-azurestack-upi	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-kube-controller-manager-operator/release-4.10/e2e-aws-ccm-install	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.9/e2e-azure-ccm-install	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-cloud-controller-manager-operator/release-4.9/e2e-azure-ccm	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/cluster-etcd-operator/release-4.9/e2e-gcp-disruptive-ovn	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/installer/release-4.10/e2e-azurestack	5082ba3	link	unknown	/test pj-rehearse
ci/rehearse/openshift/azure-disk-csi-driver-operator/release-4.10/e2e-azurestack-csi	5082ba3	link	unknown	/test pj-rehearse
ci/prow/pj-rehearse	5082ba3	link	false	/test pj-rehearse

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[vrutkovs]

There is a high probability that tests are not in TEST_SKIPS especially
for different TEST_SUITES(i.e. openshift/conformance/serial) and this
does not an indication of a failure.

Is there any example of that? Erroring on invalid TEST_SKIPS helps us keep it updated

[ardaguclu]

Thanks @vrutkovs. After further investigation, you are right. Indeed it is useful for showing possible errors.

I thought that all workflows use same TEST_SKIPS section, however it seems that they might use their own TEST_SKIPS by overriding the default one.

Thus, in favor of this, I'm closing this PR.

/close

[openshift-ci]

@ardaguclu: Closed this PR.

Details

In response to this:

Thanks @vrutkovs. After further investigation, you are right. Indeed it is useful for showing possible errors.

I thought that all workflows use same TEST_SKIPS section, however it seems that they might use their own TEST_SKIPS by overriding the default one.

Thus, in favor of this, I'm closing this PR.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.