[release-4.16] OCPBUGS-64858: Fix stale EIP assignments during failover and controller restart #2851
Conversation
pperiyasamy
changed the title
openshift-ci-robot
added
jira/severity-critical
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-64858, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/verified by @huiran0826 |
openshift-ci-robot
added
the
verified
|
@huiran0826: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Scenario: - Nodes: node-1, node-2, node-3 - Egress IPs: EIP-1 - Pods: pod1 on node-1, pod2 on node-3 (pods are created via deployment replicas) - Egress-assignable nodes: node-1, node-2 - EIP-1 assigned to node-1 During a simultaneous reboot of node-1 and node-2, EIP-1 failed over to node-2 and ovnkube-controller restarted at nearly the same time: 1) EIP-1 was reassigned to node-2 by the cluster manager. 2) The sync EIP happened for EIP1 with stale status, though it cleaned SNATs/LRPs referring to node-1 due to outdated pod IPs (this is because pods will be recreated due to node reboots). 3) pod1/pod2 Add events arrived while the informer cache still had the old EIP status, so new SNATs/LRPs were created pointing to node-1. 4) The EIP-1 Add event arrived with the new status; entries for node-2 were added/updated. 5) Result: stale SNATs and LRPs with stale nexthops for node-1 remained. Fix: - Populate pod EIP status during EgressIP sync so podAssignment has accurate egressStatuses. - Reconcile stale assignments using podAssignment (egressStatuses) when the informer cache is not up to date, ensuring SNAT/LRP for the previously assigned node are corrected. - Remove stale EIP SNAT entries for remote-zone pods accordingly. - Add coverage for simultaneous EIP failover and controller restart. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 1667a51) (cherry picked from commit 7060af6) (cherry picked from commit 0a3e1b9) (cherry picked from commit 75f0b7e) (cherry picked from commit 69afe23)
During an ovnkube-controller restart, pod add/remove events for EgressIP-served pods may occur before the factory.egressIPPod handler is registered in the watch factory. As a result, the EIP controller never able to handle pod delete, leading to stale logical router policy (LRP) entry. Scenario: ovnkube-controller starts. The EIP controller processes the namespace add event (oc.WatchEgressIPNamespaces) and creates an LRP entry for the served pod. The pod is deleted. The factory.egressIPPod handler registration happens afterward via oc.WatchEgressIPPods. The pod delete event is never processed by the EIP controller. Fix: 1. Start oc.WatchEgressIPPods followed by oc.WatchEgressIPNamespaces. 2. Sync EgressIPs before registering factory.egressIPPod event handler. 3. Removal of Sync EgressIPs for factory.EgressIPNamespaceType which is no longer needed. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 8975b00) (cherry picked from commit b8303a2) (cherry picked from commit 3f391e7) (cherry picked from commit 6cd0f09) (cherry picked from commit 0dd5bb4)
When the EIP controller cleans up a stale EIP assignment for a pod, it also removes the pod object from the podAssignment cache. This is incorrect, as it prevents the EIP controller from processing the subsequent pod delete event. Scenario: 1. pod-1 is served by eip-1, both hosted on node1. 2. node1’s ovnkube-controller restarts. 3. Pod add event is received by the EIP controller — no changes. 4. eip-1 moves from node1 to node0. 5. The EIP controller receives the eip-1 add event. 6. eip-1 cleans up pod-1’s stale assignment (SNAT and LRP) for node1, but removes the pod object from the podAssignment cache when no other assignments found. 7. The EIP controller programs the LRP entry with node0’s transit IP as the next hop, but the pod assignment cache is not updated with new podAssignmentState. 8. The pod delete event is received by the EIP controller but ignored, since the pod object is missing from the assignment cache. So this commit fixes the issue by adding podAssignmentState back into podAssignment cache at step 7. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 16dedd1) (cherry picked from commit f4e2c17) (cherry picked from commit be0f3b8) (cherry picked from commit 3a524b5) (cherry picked from commit 7d8eb17)
pperiyasamy
force-pushed
the
eip_failover_ovnkubenode_restart_4.16
branch
from
9eecc76 to
f5ffbe1
Compare
openshift-ci-robot
removed
the
verified
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-64858, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/verified by @huiran0826 |
openshift-ci-robot
added
the
verified
|
@huiran0826: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest-required |
|
/payload 4.16 ci blocking |
|
@pperiyasamy: trigger 5 job(s) of type blocking for the ci release of OCP 4.16
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1946d6f0-c0b2-11f0-81da-b18980cd49e2-0 trigger 8 job(s) of type blocking for the nightly release of OCP 4.16
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1946d6f0-c0b2-11f0-81da-b18980cd49e2-1 |
|
/retest-required |
|
/payload-job periodic-ci-openshift-release-master-ci-4.16-e2e-azure-ovn-upgrade |
|
@pperiyasamy: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1cc2b2a0-c133-11f0-9b3e-6cff4bd852a7-0 |
|
/payload-job periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-gcp-ovn-rt-upgrade |
|
@pperiyasamy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/493b92e0-c154-11f0-976e-1d7f8aac925f-0 |
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-bug
openshift-ci-robot
removed
the
jira/invalid-bug
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-64858, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/assign @jcaamano |
|
/retest-required |
|
/lgtm |
openshift-ci
bot
added
the
backport-risk-assessed
openshift-ci
bot
assigned
anuragthehatter,
asood-rh,
barbora137,
huiran0826,
jechen0648,
Meina-rh,
mffiedler,
qiowang721 and
rbbratta
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jcaamano, pperiyasamy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/retest-required |
2 similar comments
|
/retest-required |
|
/retest-required |
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/retest-required |
openshift-merge-bot
bot
merged commit 8df7e87
into
openshift:release-4.16
|
@pperiyasamy: Jira Issue Verification Checks: Jira Issue OCPBUGS-64858 Jira Issue OCPBUGS-64858 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Fix included in accepted release 4.16.0-0.nightly-2025-11-17-025450 |
14 participants
Manual cherry-pick of 4.17 PR #2850.
Resolved merge conflicts in
go-controller/pkg/ovn/default_network_controller.go(commit 0dd5bb4), specifically within the SyncFunc switch case block.The UTs in commits 1a813fa, 55c4ecc and f5ffbe1 needed update for AddressSets match.