OCPBUGS-58501,OCPBUGS-59657,OCPBUGS-42303,OCPBUGS-61566: DownStream Merge [09-09-2025]#2750
Conversation
sometimes a tmp pv image file used for setting up the runner gets leftover in /mnt. Add the removal of that in the "Free up disk space" step. Also move that step to a composite action to deduplicate it for easier maintenance Also added the Runner Diagnostics step as the first step that runs for every job Signed-off-by: Jamo Luhrsen <jluhrsen@gmail.com>
When multiple networks support was first added, all controllers that were added used the label "Secondary" to indicate they were not "Default". When UDN was added, it allowed "Secondary" networks to function as the primary network for a pod, creating terminology confusion. We now treat non-default networks all as "User-Defined Networks". This commit changes all naming to conform to the latter. The only places secondary is used now is for distinguishing whether or not a UDN is acting as a primary or secondary network for a pod (it's role). The only exception to this is udn-isolation. I did not touch this because it relies on dbIDs, which would impact functionality for upgrade. There is no functional change in this commit. Signed-off-by: Tim Rozet <trozet@nvidia.com>
Rabbit found these during code review. Signed-off-by: Tim Rozet <trozet@nvidia.com>
fix intermittent disk space issue
Moves ACL DBIDs from "Secondary" to "PrimaryUDN". Signed-off-by: Tim Rozet <trozet@nvidia.com>
Signed-off-by: Tim Rozet <trozet@nvidia.com>
Currently, we are force exiting with the trap before the background processes can end, container is removed and the orphaned processes end early causing our config to go into an unknown state because we dont end in an orderly manner. Wait until the pid file for ovnkube controller with node is removed which shows the process has completed. Signed-off-by: Martin Kennelly <mkennell@redhat.com>
Prevent ovn-controller from sending stale GARP by adding drop flows on external bridge patch ports until ovnkube-controller synchronizes the southbound database - henceforth known as "drop flows". This addresses race conditions where ovn-controller processes outdated SB DB state before ovnkube-controller updates it, particularly affecting EIP SNAT configurations attached to logical router ports. Fixes: https://issues.redhat.com/browse/FDP-1537 ovnkube-controller controls the lifecycle of the drop flows. ovs / ovn-controller running is required to configure external bridge. Downstream, the external bridge maybe precreated and ovn-controller will use this. This fix considers three primary scenarios: node, container and pod restart. On Node restart means the ovs flows installed priotior to reboot on the node are cleared but the external bridge exists. Add the flows before ovnkube controller with node starts. The reason to add it here is that our gateway code depends on ovn-controller started and running... There is now a race here between ovn-controller starting (and garping) before we set this flow but I think the risk is low however it needs serious testing. The reason I did not naturally at the drop flows before ovn-controller started is because I have no way to detect if its a node reboot or pod reboot and i dont want to inject drop flows for simple ovn-controller container restart which could disrupt traffic. ovnkube-controller starts, we create a new gateway and apply flows the same flows in-order to ensure we always drop GARP when ovnkube controller hasn't sync. Remove the flows when ovnkube-controller has syncd. There is also a race here between ovnkube-controller removing the flows and ovn-controller GARPing with stale SB DB info. There is no easy way to detect what SB DB data ovn-controller has consumed. On Pod restart, we add the drop flows before exit. ovnkube-controller-with-node will also add it before it starts the go code. Container restart: - ovnkube-controller: adds flows upon start and exit - ovn-controller: no changes While the drop flows are set, OVN may not be able to resolve IPs it doesn't know about in its Logical Router pipelines generation. Following removal of the drop flows, OVN may resolve the IPs using GARP requests. OVN-Controller always sends out GARPs with op code 1 on startup. Signed-off-by: Martin Kennelly <mkennell@redhat.com>
Fix naming of "Secondary" to be "User-Defined"
sriovnet lib contains fixes to handle local and remote VF representors co-existing. this fix is relevant for case where ovn-kubernetes runs in DPU mode and there are VFs for both the external host and local VFs (which are used on the DPU itself) Signed-off-by: adrianc <adrianc@nvidia.com>
add new netlink Route fields: MTULock, RtoMinLock Signed-off-by: adrianc <adrianc@nvidia.com>
run go mod tidy in e2e test folder Signed-off-by: adrianc <adrianc@nvidia.com>
We configure the interface in the pods via CNI, thus it doesn't make sense to configure OVN IPAM, and use it to allocate and configure the pod IP address. Signed-off-by: Miguel Duarte Barroso <mdbarroso@redhat.com>
chore: bump sriovnet lib
multi-homing, tests: do not use OVN provided IPAM in L3 nets
The node-encap-ips annotation should not be set when running in DPU host mode since the encap IP is not available on the DPU host. This prevents errors during node initialization when the encap IP cannot be determined. - Add conditional check for NodeModeDPUHost before setting encap IP annotation - Maintain existing behavior for non-DPU host modes - Fixes initialization errors in DPU host mode deployments Signed-off-by: Alin Serdean <aserdean@nvidia.com>
Fixes regression from 1448d5a The previous commit dropped matching on in_port so that localnet ports would also use table 1. This allows reply packets from a localnet pod towards the shared OVN/LOCAL IP to be sent to the correct port. However, a regression was introduced where traffic coming from these localnet ports to any destination would be sent to table 1. Egress traffic from the localnet ports is not committed to conntrack, so by sending to table=1 via CT we were getting a miss. This is especially bad for hardware offload where a localnet port is being used as the Geneve encap port. In this case all geneve traffic misses in CT lookup and is not offloaded. Table 1 is intended to be for handling IP traffic destined to the shared Gateway IP/MAC that both the Host and OVN use. It is also used to handle reply traffic for Egress IP. To fix this problem, we can add dl_dst match criteria to this flow, ensuring that only traffic destined to the Host/OVN goes to table 1. Furthermore, after fixing this problem there still exists the issue that localnet -> host/OVN egress traffic will still enter table 1 and CT miss. Potentially this can be fixed with always committing egress traffic, but it might have performance penalty, so deferring that fix to a later date. Signed-off-by: Tim Rozet <trozet@nvidia.com>
We did this for IPv4 in 1448d5a, but forgot about IPv6. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Add dl_dst=$breth0 to table=0, prio=50 for IPv6 We want to match in table=1 only conntrack'ed reply traffic whose next hop is either OVN or the host. As a consequence, localnet traffic whose next hop is an external router (and that might or might not be destined to OVN/host) should bypass table=1 and just hit the NORMAL flow in table=0. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
We already tested localnet -> host, let's also cover connections initiated from the host. The localnet uses IPs in the same subnet as the host network. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
We have two non-InterConnect CI lanes for multihoming, while only one with IC enabled (and local gw). We need coverage with IC enabled for both gateway modes, so let's make an existing non-IC lane IC enabled, set it as dualstack and gateway=shared to have better coverage. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
This is needed because we will need to generate IPs from different subnets than just the host subnet. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
The localnet is on a subnet different than the host subnet, the corresponding NAD is configured with a VLAN ID, the localnet pod uses an external router to communicate to cluster pods. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
In testing we saw how an invalid conntrack state would drop all echo requests after the first one. Let's send three pings in each test then. Signed-off-by: Riccardo Ravaioli <rravaiol@redhat.com>
|
|
|
/test 4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade .. There was also what i believe is a flake for test |
|
/override ci/prow/lint |
|
@martinkennelly: Overrode contexts on behalf of martinkennelly: ci/prow/lint DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@martinkennelly: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/override ci/prow/4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade job |
|
@martinkennelly: Overrode contexts on behalf of martinkennelly: ci/prow/4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Bug for RT upgrade job: https://issues.redhat.com/browse/OCPBUGS-62093 I see ovn in the job title so i am assigning it against ovn-k team. |
all jobs have ovn in job title, is this really a networking issue? if there is a CI issue around provisioning resources not sure what we can do, unless I amisunderstood the bug title. Better path is to search for OCP CI search sippy for failures around this? (wearing dispatch hat, not enough info to suggest any evidence its us, I'll need to close it :P unless more triage was done) |
|
Waiting for other eng to add lgtm and qe to add the necessary label |
not our problem. it's happening in the periodics as well. |
|
@jluhrsen do you know who i can reassign the bug to? |
|
/lgtm |
|
@martinkennelly: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/unhold |
openshift-ci
bot
removed
the
do-not-merge/hold
|
/lgtm on behalf of @martinkennelly |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: martinkennelly, tssurya The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/verified by jechen |
openshift-ci-robot
added
the
verified
|
@jechen0648: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-merge-bot
bot
merged commit e712193
into
openshift:master
|
@martinkennelly: Jira Issue Verification Checks: Jira Issue OCPBUGS-58501 Jira Issue OCPBUGS-58501 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 Jira Issue Verification Checks: Jira Issue OCPBUGS-59657 Jira Issue OCPBUGS-59657 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 Jira Issue OCPBUGS-42303: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged: All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-42303 has not been moved to the MODIFIED state. This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload. Jira Issue OCPBUGS-61566: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged:
All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-61566 has not been moved to the MODIFIED state. This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
no, sorry. someone would have to look deeper in to the logs to figure out what is actually causing the slow down in the jobs and assign to whatever component they figure out. |
|
Fix included in accepted release 4.21.0-0.nightly-2025-09-25-082813 |
20 participants
No description provided.