Bug 1935539: vSphere: udp tnl workaround cannot use nmcli #2482

jcpowermac · 2021-03-23T01:50:59Z

In testing this evening with openshift-sdn based install
nmcli does not change the ethtool features until the
interface has been restarted.

Switching to using ethtool directly.

In testing this evening with openshift-sdn based install nmcli does not change the ethtool features until the interface has been restarted. Switching to using ethtool directly.

openshift-ci-robot · 2021-03-23T01:51:03Z

@jcpowermac: This pull request references Bugzilla bug 1935539, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1935539: vSphere: udp tnl workaround cannot use nmcli

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcpowermac · 2021-03-23T01:57:51Z

fyi: @wking

jcpowermac · 2021-03-23T12:57:15Z

/cherry-pick release-4.7

openshift-cherrypick-robot · 2021-03-23T12:57:16Z

@jcpowermac: once the present PR merges, I will cherry-pick it on top of release-4.7 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vikaslaad · 2021-03-23T14:08:19Z

/retest

rvanderp3 · 2021-03-23T14:19:43Z

In testing this evening with openshift-sdn based install
nmcli does not change the ethtool features until the
interface has been restarted.

Switching to using ethtool directly.

I tested a clean install which incorporated this PR and confirmed that tx-udp_tnl-segmentation and tx-udp_tnl-csum-segmentation were both off on all nodes.

wking · 2021-03-23T15:34:24Z

/bugzilla refresh
/lgtm

openshift-ci-robot · 2021-03-23T15:35:31Z

@wking: This pull request references Bugzilla bug 1935539, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (huirwang@redhat.com), skipping review request.

Details

In response to this:

/bugzilla refresh
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vikaslaad · 2021-03-23T15:45:12Z

/retest

dcbw · 2021-03-23T16:25:15Z

/hold
you want to target only vmxnet3 interfaces, not every single interface on the box.

dcbw · 2021-03-23T16:27:55Z

Yes, of course when you just change the properties NM doesn't live-mirror those changes. You need to intentionally tell NM to do that, because you (as the person changing the config) know better than NM when those changes should be applied.

Have you tried nmcli dev reapply eth0 after making the nmcli connection properties changes?

dcbw · 2021-03-23T16:29:36Z

Lastly, this shouldn't be a dispatcher script. It should be an MCO systemd unit (like ovs-configuration.sh) that targets only vmxnet3 devices, and runs on system startup (or even better, before reboot!) once to disable offload property and then reapply the connection. Changes made with nmcli should be persistent.

rbbratta · 2021-03-23T16:38:42Z

To target vmxnet3

sh-4.4# nmcli -f general.driver dev show ens192
GENERAL.DRIVER:                         vmxnet3

dcbw · 2021-03-23T16:41:43Z

Something like:

#!/bin/bash
set -x
for i in $(nmcli -t -m tabular -f uuid,type,device con show --active); do
  type=$(echo ${i} | cut -d':' -f 2)
  dev=$(echo ${i} | cut -d':' -f 3)
  driver=$(nmcli -t -m tabular -f general.driver dev show ${dev})
  if [[ "${type}" == "802-3-ethernet" && "${driver}" == "vmxnet3" ]]; then
    uuid=$(echo ${i} | cut -d':' -f 1)
    nmcli con mod ${uuid} ethtool.feature-tx-udp_tnl-segmentation off
    nmcli con mod ${uuid} ethtool.feature-tx-udp_tnl-csum-segmentation off
    nmcli dev reapply ${dev}
  fi
done

jcpowermac · 2021-03-23T17:21:04Z

set -x
for i in $(nmcli -t -m tabular -f uuid,type,device con show --active); do
type=$(echo ${i} | cut -d':' -f 2)
dev=$(echo ${i} | cut -d':' -f 3)
driver=$(nmcli -t -m tabular -f general.driver dev show ${dev})
if [[ "${type}" == "802-3-ethernet" && "${driver}" == "vmxnet3" ]]; then
uuid=$(echo ${i} | cut -d':' -f 1)
nmcli con mod ${uuid} ethtool.feature-tx-udp_tnl-segmentation off
nmcli con mod ${uuid} ethtool.feature-tx-udp_tnl-csum-segmentation off
nmcli dev reapply ${dev}
fi
done

@dcbw

+ nmcli con mod 05892466-9cfa-4407-9723-4649d90e574c ethtool.feature-tx-udp_tnl-csum-segmentation off
+ nmcli dev reapply ens192
Error: Reapplying connection to device 'ens192' (/org/freedesktop/NetworkManager/Devices/3) failed: Can't reapply any changes to 'ethtool' setting
++ printf '\033k%s@%s:%s\033\' root ip-172-31-245-109 /usr/local/bin

rbbratta · 2021-03-23T17:26:43Z

might be easier to add the ethtool to configure-ovs.sh maybe, even though it is vmxnet3 specific.

vikaslaad · 2021-03-23T18:02:52Z

/retest

rbbratta · 2021-03-23T21:35:35Z

nmcli con mod 05892466-9cfa-4407-9723-4649d90e574c ethtool.feature-tx-udp_tnl-csum-segmentation off

nmcli dev reapply ens192
Error: Reapplying connection to device 'ens192' (/org/freedesktop/NetworkManager/Devices/3) failed: Can't reapply any changes to 'ethtool' setting

Maybe we can ignore this error. If we run this once on boot via systemd, then it shouldn't re-run, maybe.

openshift-bot · 2021-03-25T15:00:58Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-25T15:29:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

wking · 2021-03-25T16:29:13Z

last update job only failed [sig-network] pods should successfully create sandboxes by other on 2 failures to create the sandbox, which is not unique to this PR. And it's an AWS-specific job. I think we can /override ci/prow/e2e-agnostic-upgrade.

openshift-bot · 2021-03-25T16:31:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

kikisdeliveryservice · 2021-03-25T16:42:16Z

Letting this last run retry, will check in after.

dcbw · 2021-03-25T17:56:48Z

running out of leases again for GCP for the agnostic-upgrade job:

2021/03/25 17:02:23 error: Failed to acquire resource, current capacity: 0 free, 70 leased

dcbw · 2021-03-25T17:57:03Z

/retest

jcpowermac · 2021-03-25T19:52:47Z

@kikisdeliveryservice would you be ok with overriding now?

kikisdeliveryservice · 2021-03-25T20:00:12Z

Given the severity of the bug, the fact that this PR changes aren't implicated in the job failure, and the red-ness of the job. We can make a one time exception and override. 😄

/override ci/prow/e2e-agnostic-upgrade

openshift-ci-robot · 2021-03-25T20:01:34Z

@kikisdeliveryservice: Overrode contexts on behalf of kikisdeliveryservice: ci/prow/e2e-agnostic-upgrade

Details

In response to this:

Given the severity of the bug, the fact that this PR changes aren't implicated in the job failure, and the red-ness of the job. We can make a one time exception and override. 😄

/override ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-26T02:03:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T03:21:10Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T04:52:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T05:57:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T06:10:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T06:23:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T09:25:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

davidkarlsen · 2021-03-26T09:59:36Z

lol, this must be the most built 5 lines of codes in history :-)

jcpowermac · 2021-03-26T14:42:11Z

/skip

openshift-bot · 2021-03-26T15:03:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-03-26T17:26:40Z

@jcpowermac: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-workers-rhel7	`661bcd8`	link	`/test e2e-aws-workers-rhel7`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-03-26T18:05:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2021-03-26T22:21:57Z

@jcpowermac: All pull requests linked via external trackers have merged:

Bugzilla bug 1935539 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1935539: vSphere: udp tnl workaround cannot use nmcli

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2021-03-26T22:22:37Z

@jcpowermac: #2482 failed to apply on top of branch "release-4.7":

Applying: vSphere: udp tnl workaround cannot use nmcli
Using index info to reconstruct a base tree...
A	templates/common/vsphere/files/vsphere-disable-vmxnet3v4-features.yaml
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): templates/common/vsphere/files/vsphere-disable-vmxnet3v4-features.yaml deleted in HEAD and modified in vSphere: udp tnl workaround cannot use nmcli. Version vSphere: udp tnl workaround cannot use nmcli of templates/common/vsphere/files/vsphere-disable-vmxnet3v4-features.yaml left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 vSphere: udp tnl workaround cannot use nmcli
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-4.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

backport PR: openshift#2482 There is an issue (tbd) with the VMXNET3 v4 driver causing network traffic between various openshift services to be dropped. With the addition of VMXNET3 v4 in RHCOS 4.7 (RHEL 8.3 kernel) this workaround needs to be in place for customers running on vSphere 6.7, OpenShift 4.7 and virtual hardware version 14 or higher.

vSphere: udp tnl workaround cannot use nmcli

3cb47ec

In testing this evening with openshift-sdn based install nmcli does not change the ethtool features until the interface has been restarted. Switching to using ethtool directly.

openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Mar 23, 2021

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Mar 23, 2021

openshift-ci-robot requested review from patrickdillon and sinnykumari March 23, 2021 01:51

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 23, 2021

zhaozhanqi mentioned this pull request Mar 23, 2021

Bug 1941246: vSphere: Disable tx udp_csum segmentation #2475

Closed

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Mar 23, 2021

openshift-ci-robot assigned wking Mar 23, 2021

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 23, 2021

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 23, 2021

openshift-merge-robot merged commit 3915a1a into openshift:master Mar 26, 2021

jcpowermac mentioned this pull request Mar 29, 2021

Bug 1941246: vSphere: Disable tx-udp-tnl offload #2495

Merged

jomeier mentioned this pull request Apr 12, 2021

Router connectivity issue with OpenShiftSDN in 4.6 okd-project/okd#430

Closed

jcpowermac mentioned this pull request Apr 30, 2021

Bug 1952358: vSphere platform none vmxnet3v4 fix #2558

Closed

bo0ts mentioned this pull request Apr 15, 2023

Pod to Pod Communcation severely degraded in 4.11 on vSphere okd-project/okd#1550

Closed

Bug 1935539: vSphere: udp tnl workaround cannot use nmcli #2482

Bug 1935539: vSphere: udp tnl workaround cannot use nmcli #2482

Uh oh!

Conversation

jcpowermac commented Mar 23, 2021

Uh oh!

openshift-ci-robot commented Mar 23, 2021

Uh oh!

jcpowermac commented Mar 23, 2021

Uh oh!

jcpowermac commented Mar 23, 2021

Uh oh!

openshift-cherrypick-robot commented Mar 23, 2021

Uh oh!

vikaslaad commented Mar 23, 2021

Uh oh!

rvanderp3 commented Mar 23, 2021

Uh oh!

wking commented Mar 23, 2021

Uh oh!

openshift-ci-robot commented Mar 23, 2021

Uh oh!

vikaslaad commented Mar 23, 2021

Uh oh!

dcbw commented Mar 23, 2021

Uh oh!

dcbw commented Mar 23, 2021

Uh oh!

dcbw commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbbratta commented Mar 23, 2021

Uh oh!

dcbw commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcpowermac commented Mar 23, 2021

Uh oh!

rbbratta commented Mar 23, 2021

Uh oh!

vikaslaad commented Mar 23, 2021

Uh oh!

rbbratta commented Mar 23, 2021

Uh oh!

openshift-bot commented Mar 25, 2021

Uh oh!

openshift-bot commented Mar 25, 2021

Uh oh!

wking commented Mar 25, 2021

Uh oh!

openshift-bot commented Mar 25, 2021

Uh oh!

kikisdeliveryservice commented Mar 25, 2021

Uh oh!

dcbw commented Mar 25, 2021

Uh oh!

dcbw commented Mar 25, 2021

Uh oh!

jcpowermac commented Mar 25, 2021

Uh oh!

kikisdeliveryservice commented Mar 25, 2021

Uh oh!

openshift-ci-robot commented Mar 25, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

openshift-bot commented Mar 26, 2021

Uh oh!

davidkarlsen commented Mar 26, 2021

Uh oh!

dcbw commented Mar 23, 2021 •

edited

Loading

dcbw commented Mar 23, 2021 •

edited

Loading

openshift-ci bot commented Mar 26, 2021 •

edited

Loading