Bug 1880591: 4.6 Use ovs-configuration file to determine if OVS is running in systemd by trozet · Pull Request #825 · openshift/cluster-network-operator

trozet · 2020-10-05T14:44:52Z

Checking files or symlinks on disks seems to be flaky and there are time
windows where files are laid down on the disk but the running version is
still 4.5. This changes the logic to determine if OVS is running on the
host by checking the for a file generated when ovs-configuration is
executed.

Signed-off-by: Tim Rozet trozet@redhat.com

openshift-ci-robot · 2020-10-05T14:44:54Z

@trozet: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

[WIP] DNM 4.6 check Use OS version to determine if OVS is running in systemd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2020-10-05T14:46:56Z

/test e2e-gcp-ovn

openshift-ci-robot · 2020-10-05T14:47:13Z

@trozet: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test e2e-aws-multitenant
/test e2e-aws-ovn
/test e2e-aws-sdn-multi
/test e2e-aws-sdn-single
/test e2e-azure-ovn
/test e2e-gcp
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi
/test e2e-operator-with-custom-vxlan-port
/test e2e-ovn-hybrid-step-registry
/test e2e-ovn-step-registry
/test e2e-upgrade
/test e2e-vsphere-ovn
/test e2e-windows-hybrid-network
/test images
/test unit
/test verify

Use /test all to run the following jobs:

pull-ci-openshift-cluster-network-operator-release-4.6-e2e-aws-sdn-multi
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-aws-sdn-single
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-azure-ovn
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-gcp
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-metal-ipi
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-operator-with-custom-vxlan-port
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-ovn-hybrid-step-registry
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-ovn-step-registry
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-upgrade
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-vsphere-ovn
pull-ci-openshift-cluster-network-operator-release-4.6-e2e-windows-hybrid-network
pull-ci-openshift-cluster-network-operator-release-4.6-images
pull-ci-openshift-cluster-network-operator-release-4.6-unit
pull-ci-openshift-cluster-network-operator-release-4.6-verify

Details

In response to this:

/test e2e-gcp-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2020-10-05T14:47:28Z

/test e2e-gcp-ovn-upgrade

trozet · 2020-10-05T16:06:28Z

cluster-botAPP 11:42 AM
job test upgrade 4.5.0-0.nightly #825 aws,ovn succeeded

cluster-botAPP 11:50 AM
job test upgrade 4.5.0-0.nightly #825 aws succeeded

cluster-botAPP 12:01 PM
job test upgrade 4.5.0-0.nightly #825 gcp,ovn succeeded
12:06
job test upgrade 4.5.0-0.nightly #825 gcp succeeded

juanluisvaladas · 2020-10-05T16:27:19Z

/lgtm

openshift-ci-robot · 2020-10-05T16:28:17Z

@trozet: This pull request references Bugzilla bug 1880591, which is invalid:

expected Bugzilla bug 1880591 to depend on a bug targeting a release in 4.7.0 and in one of the following states: MODIFIED, VERIFIED, but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1880591: 4.6 check Use OS version to determine if OVS is running in systemd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2020-10-05T16:55:02Z

/bugzilla refresh

openshift-ci-robot · 2020-10-05T16:55:11Z

@trozet: This pull request references Bugzilla bug 1880591, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1885344 is in the state VERIFIED, which is one of the valid states (MODIFIED, VERIFIED)
dependent Bugzilla bug 1885344 targets the "4.7.0" release, which is one of the valid target releases: 4.7.0
bug has dependents

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trozet · 2020-10-05T17:00:45Z

/retest

trozet · 2020-10-05T20:23:08Z

/retest

trozet · 2020-10-05T22:46:33Z

/retest

openshift-bot · 2020-10-05T23:24:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-10-05T23:37:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-10-05T23:50:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

trozet · 2020-10-06T00:43:47Z

vsphere and hybrid step are only failing:
Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured

which is being fixed by #824

trozet · 2020-10-06T00:44:01Z

/override ci/prow/e2e-ovn-hybrid-step-registry

trozet · 2020-10-06T00:44:11Z

/override ci/prow/e2e-vsphere-ovn

trozet · 2020-10-07T13:09:13Z

/retest

danwinship · 2020-10-07T14:14:29Z

/hold cancel
/lgtm

openshift-ci-robot · 2020-10-07T14:14:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, juanluisvaladas, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [danwinship,trozet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

trozet · 2020-10-07T14:21:34Z

/hold

until I can verify job logs and cluster bot upgrades

trozet · 2020-10-07T17:17:16Z

logs look ok to me

/hold cancel

trozet · 2020-10-07T17:17:27Z

/retest

openshift-bot · 2020-10-07T19:33:05Z

/retest

Please review the full test history for this PR and help us cut down flakes.

trozet · 2020-10-07T19:47:25Z

@juanluisvaladas @tssurya I'm seeing vsphere failed because:

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/825/pull-ci-openshift-cluster-network-operator-release-4.6-e2e-vsphere-ovn/1313891263669342208

fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"TargetDown\",\"alertstate\":\"firing\",\"job\":\"ovnkube-master\",\"namespace\":\"openshift-ovn-kubernetes\",\"service\":\"ovn-kubernetes-master\",\"severity\":\"warning\"},\"value\":[1602096410.162,\"1\"]}]"

I don't see ovnkube master container go down. The value here looks suspicious:[1602096410.162,"1"] but I don't know anything about prom alerts. This PR run included the fix for #826

trozet · 2020-10-07T19:54:02Z

vsphere passed all tests and is running in systemd OVS. The alert was the only thing it failed...overriding

/override ci/prow/e2e-vsphere-ovn

openshift-ci-robot · 2020-10-07T19:54:23Z

@trozet: Overrode contexts on behalf of trozet: ci/prow/e2e-vsphere-ovn

Details

In response to this:

vsphere passed all tests and is running in systemd OVS. The alert was the only thing it failed...overriding

/override ci/prow/e2e-vsphere-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tssurya · 2020-10-07T20:16:14Z

I don't see ovnkube master container go down. The value here looks suspicious:[1602096410.162,"1"] but I don't know anything about prom alerts. This PR run included the fix for #826

This alert's value is indeed strange.. usually it doesn't exceed 100 since we take a percentage of the value. So for example since we have 3 ovnkube-master pods and say one of the metrics port is not reachable, then 1/3*100 = 33.33 would be the value. I have no idea how the value here is so huge :D

vrutkovs · 2020-10-07T20:20:42Z

First is a timestamp, the second one is value. I'm seeing similar alerts in OKD nightly runs, seems started at around Oct 3 and flaking pretty often

tssurya · 2020-10-07T21:09:53Z

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/825/pull-ci-openshift-cluster-network-operator-release-4.6-e2e-vsphere-ovn/1313891263669342208
fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"TargetDown\",\"alertstate\":\"firing\",\"job\":\"ovnkube-master\",\"namespace\":\"openshift-ovn-kubernetes\",\"service\":\"ovn-kubernetes-master\",\"severity\":\"warning\"},\"value\":[1602096410.162,\"1\"]}]"
I don't see ovnkube master container go down. The value here looks suspicious:[1602096410.162,"1"] but I don't know anything about prom alerts. This PR run included the fix for #826

For this particular run, there are clearly some issues:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/825/pull-ci-openshift-cluster-network-operator-release-4.6-e2e-vsphere-ovn/1313891263669342208/artifacts/e2e-vsphere-ovn/gather-extra/pods/openshift-ovn-kubernetes_ovnkube-master-62rxw_ovnkube-master.log

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-network-operator/825/pull-ci-openshift-cluster-network-operator-release-4.6-e2e-vsphere-ovn/1313891263669342208/artifacts/e2e-vsphere-ovn/gather-extra/pods/openshift-ovn-kubernetes_ovnkube-master-62rxw_kube-rbac-proxy.log

Although I don't see this alert in previous runs of this job.

tssurya · 2020-10-07T21:11:23Z

First is a timestamp, the second one is value. I'm seeing similar alerts in OKD nightly runs, seems started at around Oct 3 and flaking pretty often

This should have got fixed with #826

openshift-bot · 2020-10-07T21:43:01Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-10-08T00:19:02Z

/retest

Please review the full test history for this PR and help us cut down flakes.

trozet · 2020-10-08T02:07:39Z

/override ci/prow/e2e-vsphere-ovn

openshift-ci-robot · 2020-10-08T02:07:59Z

@trozet: Overrode contexts on behalf of trozet: ci/prow/e2e-vsphere-ovn

Details

In response to this:

/override ci/prow/e2e-vsphere-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-10-08T02:12:50Z

@trozet: All pull requests linked via external trackers have merged:

Bugzilla bug 1880591 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1880591: 4.6 Use ovs-configuration file to determine if OVS is running in systemd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2020-10-10T08:52:05Z

bindata/network/openshift-sdn/sdn-ovs.yaml


          # Check to see if ovs is provided by the node:
-          if [[ -L '/host/etc/systemd/system/network-online.target.wants/ovs-configuration.service' ]]; then
+          if [ -f /host/var/run/ovs-config-executed ]; then


Where does this file gets generated? I don't see it being created by MCO

openshift/machine-config-operator#2140

only in release-4.6 branch
https://github.com/openshift/machine-config-operator/blob/release-4.6/templates/common/_base/files/configure-ovs-network.yaml#L9

in 4.7 we should only run host OVS, so I think the plan is to remove this if.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 5, 2020

openshift-ci-robot requested review from aojea and danwinship October 5, 2020 14:45

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2020

openshift-ci-robot assigned juanluisvaladas Oct 5, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 5, 2020

trozet changed the title ~~[WIP] DNM 4.6 check Use OS version to determine if OVS is running in systemd~~ Bug 1880591: 4.6 check Use OS version to determine if OVS is running in systemd Oct 5, 2020

openshift-ci-robot added bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Oct 5, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 5, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Oct 5, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 5, 2020

openshift-ci-robot mentioned this pull request Oct 7, 2020

Bug 1880591: Make ovs-configuration write a file openshift/machine-config-operator#2140

Merged

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 7, 2020

openshift-ci-robot assigned danwinship Oct 7, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 7, 2020

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 7, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 7, 2020

openshift-merge-robot merged commit 893d3e9 into openshift:release-4.6 Oct 8, 2020

openshift-ci-robot mentioned this pull request Oct 9, 2020

Bug 1880591: Fix service sync reject ACL name matching openshift/ovn-kubernetes#305

Merged

vrutkovs reviewed Oct 10, 2020

View reviewed changes

This was referenced Oct 27, 2020

Bug 1880591: Adds missing check for ovs-config-executed #836

Merged

Bug 1880591: Fixes race during 4.5->4.6 upgrade with ovn node and master openshift/ovn-kubernetes#307

Merged

Conversation

trozet commented Oct 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

openshift-ci-robot commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

juanluisvaladas commented Oct 5, 2020

Uh oh!

openshift-ci-robot commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

openshift-ci-robot commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

trozet commented Oct 5, 2020

Uh oh!

openshift-bot commented Oct 5, 2020

Uh oh!

openshift-bot commented Oct 5, 2020

Uh oh!

openshift-bot commented Oct 5, 2020

Uh oh!

trozet commented Oct 6, 2020

Uh oh!

trozet commented Oct 6, 2020

Uh oh!

trozet commented Oct 6, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

danwinship commented Oct 7, 2020

Uh oh!

openshift-ci-robot commented Oct 7, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

openshift-bot commented Oct 7, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

trozet commented Oct 7, 2020

Uh oh!

openshift-ci-robot commented Oct 7, 2020

Uh oh!

tssurya commented Oct 7, 2020

Uh oh!

vrutkovs commented Oct 7, 2020

Uh oh!

tssurya commented Oct 7, 2020

Uh oh!

tssurya commented Oct 7, 2020

Uh oh!

openshift-bot commented Oct 7, 2020

Uh oh!

openshift-bot commented Oct 8, 2020

Uh oh!

trozet commented Oct 8, 2020

Uh oh!

openshift-ci-robot commented Oct 8, 2020

Uh oh!

openshift-ci-robot commented Oct 8, 2020

Uh oh!

vrutkovs Oct 10, 2020

Choose a reason for hiding this comment

trozet commented Oct 5, 2020 •

edited

Loading