Skip to content

[WIP] CORENET-59657: allow default network -> localnet on the same node for any localnet subnet#2726

Closed
ricky-rav wants to merge 3 commits intoopenshift:masterfrom
ricky-rav:CORENET-59657_420
Closed

[WIP] CORENET-59657: allow default network -> localnet on the same node for any localnet subnet#2726
ricky-rav wants to merge 3 commits intoopenshift:masterfrom
ricky-rav:CORENET-59657_420

Conversation

@ricky-rav
Copy link
Contributor

@ricky-rav ricky-rav commented Aug 18, 2025

Testing upstream PR ovn-kubernetes/ovn-kubernetes#5480 on downstream CI

@openshift-ci-robot
Copy link
Contributor

@ricky-rav: No Jira issue with key CORENET-59657 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 18, 2025
@openshift-ci-robot
Copy link
Contributor

@ricky-rav: No Jira issue with key CORENET-59657 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

Details

In response to this:

Testing upstream PR ovn-kubernetes/ovn-kubernetes#5480 on downstream CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 18, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ricky-rav
Once this PR has been reviewed and has the lgtm label, please assign martinkennelly for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 20, 2025
@tssurya
Copy link
Contributor

tssurya commented Aug 21, 2025

/test

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 21, 2025

@tssurya: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test 4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade
/test 4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
/test 4.20-upgrade-from-stable-4.19-images
/test e2e-aws-ovn
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-hypershift
/test e2e-aws-ovn-local-gateway
/test e2e-aws-ovn-local-to-shared-gateway-mode-migration
/test e2e-aws-ovn-serial
/test e2e-aws-ovn-shared-to-local-gateway-mode-migration
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-local-gateway
/test e2e-aws-ovn-windows
/test e2e-azure-ovn-upgrade
/test e2e-gcp-ovn
/test e2e-gcp-ovn-techpreview
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-bgp
/test e2e-metal-ipi-ovn-dualstack-bgp-local-gw
/test e2e-metal-ipi-ovn-ipv6
/test gofmt
/test images
/test lint
/test okd-scos-images
/test qe-perfscale-payload-control-plane-6nodes
/test unit

The following commands are available to trigger optional jobs:

/test 4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade-ipsec
/test e2e-agent-compact-ipv4
/test e2e-aws-ovn-clusternetwork-cidr-expansion
/test e2e-aws-ovn-fdp-qe
/test e2e-aws-ovn-hypershift-conformance-techpreview
/test e2e-aws-ovn-hypershift-kubevirt
/test e2e-aws-ovn-serial-ipsec
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-techpreview
/test e2e-aws-ovn-upgrade-ipsec
/test e2e-azure-ovn
/test e2e-azure-ovn-techpreview
/test e2e-metal-ipi-ovn-bgp-virt-dualstack
/test e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-dualstack-local-gateway-techpreview
/test e2e-metal-ipi-ovn-dualstack-techpreview
/test e2e-metal-ipi-ovn-ipv4
/test e2e-metal-ipi-ovn-ipv6-techpreview
/test e2e-metal-ipi-ovn-techpreview
/test e2e-openstack-ovn
/test e2e-ovn-hybrid-step-registry
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-techpreview
/test e2e-vsphere-windows
/test okd-scos-e2e-aws-ovn
/test openshift-e2e-gcp-ovn-techpreview-upgrade
/test ovncore-perfscale-aws-ovn-large-cluster-density-v2
/test ovncore-perfscale-aws-ovn-large-node-density-cni
/test ovncore-perfscale-aws-ovn-xlarge-cluster-density-v2
/test ovncore-perfscale-aws-ovn-xlarge-node-density-cni
/test perfscale-aws-ovn-medium-cluster-density-v2
/test perfscale-aws-ovn-medium-node-density-cni
/test perfscale-aws-ovn-small-cluster-density-v2
/test perfscale-aws-ovn-small-node-density-cni
/test qe-perfscale-aws-ovn-small-udn-density-churn-l3
/test qe-perfscale-aws-ovn-small-udn-density-l2
/test qe-perfscale-aws-ovn-small-udn-density-l3
/test security

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade
pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade-ipsec
pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-images
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-edge-zones
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-hypershift
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-hypershift-conformance-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-hypershift-kubevirt
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-gateway
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-serial
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-serial-ipsec
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-shared-to-local-gateway-mode-migration
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-single-node-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-upgrade
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-upgrade-ipsec
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-upgrade-local-gateway
pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-windows
pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn
pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn-upgrade
pull-ci-openshift-ovn-kubernetes-master-e2e-gcp-ovn
pull-ci-openshift-ovn-kubernetes-master-e2e-gcp-ovn-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-bgp
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-bgp-local-gw
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-local-gateway-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-ipv6-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-techpreview
pull-ci-openshift-ovn-kubernetes-master-e2e-openstack-ovn
pull-ci-openshift-ovn-kubernetes-master-e2e-ovn-hybrid-step-registry
pull-ci-openshift-ovn-kubernetes-master-e2e-vsphere-ovn
pull-ci-openshift-ovn-kubernetes-master-e2e-vsphere-ovn-techpreview
pull-ci-openshift-ovn-kubernetes-master-gofmt
pull-ci-openshift-ovn-kubernetes-master-images
pull-ci-openshift-ovn-kubernetes-master-lint
pull-ci-openshift-ovn-kubernetes-master-okd-scos-e2e-aws-ovn
pull-ci-openshift-ovn-kubernetes-master-okd-scos-images
pull-ci-openshift-ovn-kubernetes-master-openshift-e2e-gcp-ovn-techpreview-upgrade
pull-ci-openshift-ovn-kubernetes-master-qe-perfscale-aws-ovn-small-udn-density-churn-l3
pull-ci-openshift-ovn-kubernetes-master-qe-perfscale-aws-ovn-small-udn-density-l3
pull-ci-openshift-ovn-kubernetes-master-qe-perfscale-payload-control-plane-6nodes
pull-ci-openshift-ovn-kubernetes-master-security
pull-ci-openshift-ovn-kubernetes-master-unit
Details

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tssurya
Copy link
Contributor

tssurya commented Aug 21, 2025

/payload-job periodic-ci-openshift-ovn-kubernetes-release-4.20-periodics-downstream-merge

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 21, 2025

@tssurya: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-ovn-kubernetes-release-4.20-periodics-downstream-merge

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1d1dfea0-7e6f-11f0-8461-dc5c126c2b8c-0

@tssurya
Copy link
Contributor

tssurya commented Aug 25, 2025

/payload-job periodic-ci-openshift-ovn-kubernetes-release-4.20-periodics-downstream-merge

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 25, 2025

@tssurya: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-ovn-kubernetes-release-4.20-periodics-downstream-merge

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5e7ff7a0-81d4-11f0-8d78-147461d9bb45-0

@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2025
@ricky-rav ricky-rav closed this Aug 29, 2025
@ricky-rav ricky-rav reopened this Aug 29, 2025
@openshift-ci-robot
Copy link
Contributor

@ricky-rav: No Jira issue with key CORENET-59657 exists in the tracker at https://issues.redhat.com/.
Once a valid jira issue is referenced in the title of this pull request, request a refresh with /jira refresh.

Details

In response to this:

Testing upstream PR ovn-kubernetes/ovn-kubernetes#5480 on downstream CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

trozet and others added 3 commits September 1, 2025 09:07
Fixes regression from 1448d5a

The previous commit dropped matching on in_port so that localnet ports
would also use table 1. This allows reply packets from a localnet pod
towards the shared OVN/LOCAL IP to be sent to the correct port.

However, a regression was introduced where traffic coming from these
localnet ports to any destination would be sent to table 1. Egress
traffic from the localnet ports is not committed to conntrack, so by
sending to table=1 via CT we were getting a miss.

This is especially bad for hardware offload where a localnet port is
being used as the Geneve encap port. In this case all geneve traffic
misses in CT lookup and is not offloaded.

Table 1 is intended to be for handling IP traffic destined to the shared
Gateway IP/MAC that both the Host and OVN use. It is also used to handle
reply traffic for Egress IP. To fix this problem, we can add dl_dst
match criteria to this flow, ensuring that only traffic destined to the
Host/OVN goes to table 1.

Furthermore, after fixing this problem there still exists the issue that
localnet -> host/OVN egress traffic will still enter table 1 and CT
miss. Potentially this can be fixed with always committing egress
traffic, but it might have performance penalty, so deferring that fix to
a later date.

Signed-off-by: Tim Rozet <trozet@nvidia.com>
We did this for IPv4 in 1448d5a, but forgot about IPv6.

Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Add dl_dst=$breth0 to table=0, prio=50 for IPv6

We want to match in table=1 only conntrack'ed reply traffic whose next hop is either OVN or the host. As a consequence, localnet traffic whose next hop is an external router (and that might or might not be destined to OVN/host) should bypass table=1 and just hit the NORMAL flow in table=0.

Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 1, 2025

@ricky-rav: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure-ovn-upgrade 493bd87 link true /test e2e-azure-ovn-upgrade
ci/prow/e2e-gcp-ovn-techpreview 493bd87 link true /test e2e-gcp-ovn-techpreview
ci/prow/e2e-aws-ovn-serial-ipsec 493bd87 link false /test e2e-aws-ovn-serial-ipsec
ci/prow/e2e-aws-ovn-hypershift-kubevirt 493bd87 link false /test e2e-aws-ovn-hypershift-kubevirt
ci/prow/4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade 493bd87 link true /test 4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
ci/prow/security 493bd87 link false /test security
ci/prow/e2e-openstack-ovn 493bd87 link false /test e2e-openstack-ovn
ci/prow/lint 493bd87 link true /test lint
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp-local-gw 493bd87 link true /test e2e-metal-ipi-ovn-dualstack-bgp-local-gw
ci/prow/e2e-aws-ovn-hypershift-conformance-techpreview 493bd87 link false /test e2e-aws-ovn-hypershift-conformance-techpreview
ci/prow/e2e-metal-ipi-ovn-dualstack-bgp 493bd87 link true /test e2e-metal-ipi-ovn-dualstack-bgp

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ricky-rav
Copy link
Contributor Author

/payload-aggregate help

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 1, 2025

@ricky-rav: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

@ricky-rav
Copy link
Contributor Author

/payload 4.20 ci blocking
/payload 4.20 nightly blocking

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 1, 2025

@ricky-rav: trigger 5 job(s) of type blocking for the ci release of OCP 4.20

  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aks
  • periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e60a3480-874f-11f0-8d8c-7218c8341668-0

trigger 10 job(s) of type blocking for the nightly release of OCP 4.20

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node
  • periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-upgrade-fips
  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
  • periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn-conformance
  • periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-serial
  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview
  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-ovn-bm
  • periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-ovn-ipv6

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e60a3480-874f-11f0-8d8c-7218c8341668-1

@ricky-rav
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 2, 2025

@ricky-rav: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

@ricky-rav
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade 5
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade 5
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview 5

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 2, 2025

@ricky-rav: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/54dca7e0-87db-11f0-8486-2525d5d4339c-0

@ricky-rav
Copy link
Contributor Author

/retest

@tssurya
Copy link
Contributor

tssurya commented Sep 3, 2025

/retest-required

@ricky-rav
Copy link
Contributor Author

The OVS CPU consumption looks in line with the values we saw in the past, with no increase with respect to our baseline.

In a previous PR I observed that values were in the 650-750 range for ovs_vswitchd_process_cpu_seconds_total:
ovn-kubernetes/ovn-kubernetes#5313 (comment)

Similarly, I took the first four jobs for aggregator-periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-upgrade-fips (https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregator-periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-upgrade-fips/1962551908463808512) and found that ovs_vswitchd_process_cpu_seconds_total is within the same range:
https://drive.google.com/drive/folders/1TjZudKz3xMwADu-oPL_gm5QzcSr3Gah_?usp=drive_link

@tssurya
Copy link
Contributor

tssurya commented Sep 3, 2025

@ricky-rav: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/54dca7e0-87db-11f0-8486-2525d5d4339c-0

Verifying the failures:

Starting with gcp-ovn-upgrade

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregator-periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade/1962803072916787200

2/5 passed, 3 failed:

  1. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-e2e-gcp-ovn-upgrade/1962802961646096384
: [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort service [Suite:openshift/conformance/parallel] [Suite:k8s] expand_less	1m11s
{  fail [k8s.io/kubernetes/test/e2e/network/conntrack.go:211]: Failed to connect to backend 2}

is known bug tracked here: https://issues.redhat.com/browse/OCPBUGS-59496

  1. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-e2e-gcp-ovn-upgrade/1962802961146974208

Pod in place resize issues is unrelated

  1. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-e2e-gcp-ovn-upgrade/1962802960152924160

@ricky-rav I don't know what this is:

: [sig-network] Networking IPerf2 [Feature:Networking-Performance] should run iperf2 [Suite:openshift/conformance/parallel] [Suite:k8s] expand_less	28s
{  fail [k8s.io/kubernetes/test/e2e/framework/pod/exec_util.go:113]: failed to execute command in pod iperf2-clients-nf5wp, container app: error sending request: Post "https://api.ci-op-vwctby6r-e9624.XXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/e2e-network-perf-2414/pods/iperf2-clients-nf5wp/exec?command=%2Fbin%2Fsh&command=-c&command=iperf+-v+%7C%7C+true&container=app&stderr=true&stdout=true": read tcp 10.131.219.60:56716->34.49.250.128:6443: read: connection reset by peer: error sending request: Post "https://api.ci-op-vwctby6r-e9624.XXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/e2e-network-perf-2414/pods/iperf2-clients-nf5wp/exec?command=%2Fbin%2Fsh&command=-c&command=iperf+-v+%7C%7C+true&container=app&stderr=true&stdout=true": read tcp 10.131.219.60:56716->34.49.250.128:6443: read: connection reset by peer}

is this actually happening in other lanes?

gcp-ovn-upgrade-rt variant

rt is also struggling on presubmits: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_ovn-kubernetes/2726/pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962411971949105152 and https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_ovn-kubernetes/2726/pull-ci-openshift-ovn-kubernetes-master-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962803114826272768 both unrelated

for periodics 5/5 failed which is super bad

  1. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962802965521633280 - unrelated testproject API
  2. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962802966553432064 - this is very BAD @ricky-rav PTAL: : [sig-architecture] platform pods in ns/openshift-ovn-kubernetes should not exit an excessive amount of times looks suspicious
  3. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962802964200427520 - unrelated
  4. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962802963768414208 - unrelated
  5. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-ovn-kubernetes-2726-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade/1962802968034021376 - In place resize thingy which is unrelated

@ricky-rav please do the same for the 3rd periodics i.e AWS

@tssurya
Copy link
Contributor

tssurya commented Sep 3, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade 5
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade 5
/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview 5

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 3, 2025

@tssurya: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade
  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/075faf80-88a8-11f0-86d5-0a400830c660-0

@ricky-rav ricky-rav closed this Sep 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants