route entry would disappear after the LoadBalancer is registered #4361

yuzs2 · 2022-11-01T10:15:03Z

Describe the bug
When creating a LoadBalancer type service, one route entry would be registered on the node ip route. However, in some cases, we couldn't found the route entry after the LoadBalancer is created.

To Reproduce
This issue can be intermittently reproduced.
For example, we created below three LoadBalancers:

kubectl get svc                    
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
loadbalancer-cluster-nh-000                    LoadBalancer   172.21.36.62     10.78.144.51                               40001:21766/TCP                              9s
loadbalancer-cluster-nh-111                    LoadBalancer   172.21.74.217    10.78.144.49                               40005:22285/TCP                              5m48s
loadbalancer-cluster-nh-222                    LoadBalancer   172.21.83.97     10.78.144.48                               40004:31250/TCP                              60s

However, we could only find 2 route entries on the node:

Every 1.0s: ip route | grep 144                                                                 unbuntu-minion001: Tue Nov  1 10:07:29 2022

10.78.144.51 via 169.254.0.253 dev gw0
10.78.144.49 via 169.254.0.253 dev gw0

The "10.78.144.48" just disappeared. As a result, when we deleted the LoadBalancer "loadbalancer-cluster-nh-222", we would receive a flood of error message:

E1101 09:22:59.491998       1 proxier.go:165] "Failed to remove flows and configurations of Service" err="failed to remove Service LoadBalancer traffic redirecting flows: failed to delete routing entry for LoadBalancer ingress IP 10.78.144.50: no such process" Service="loadbalancer-cluster-nh-222:"

These Antrea errors are producing over 60% of all logging traffic in our clusters

Expected
Please identify the root cause and fix the issue.

Versions:
Please provide the following information:

Antrea version (Docker image tag): antrea/antrea-ubuntu:v1.5.0
Kubernetes version (use kubectl version): 1.19.9
Container runtime: which runtime are you using (e.g. containerd, cri-o, docker) and which version are you using? docker 19.03.14

The text was updated successfully, but these errors were encountered:

tnqn · 2022-11-08T15:37:13Z

Hi @yuzs2 @hongliangl Did we ever see the route for "10.78.144.48" on that Node? Want to confirm if it disappeared or it may be never installed.

hongliangl · 2022-11-08T15:38:37Z

I have saw that it has been installed and uninstalled.

tnqn · 2022-11-08T16:10:12Z

Did this happen to one Node only or all Nodes? Any agent log collected when the issue happened?

yuzs2 · 2022-11-09T05:50:43Z

This is happening on all nodes. We can see the flood of Antrea error in our log system everywhere.

tnqn · 2022-11-10T13:26:18Z

Not sure if @hongliangl manages to reproduce it, but I wonder if the issue is caused by this scenario:
LoadBalancer Service A uses 10.78.144.48 as external IP in the beginning;
LoadBalancer Service A is deleted and the external IP is released, meanwhile LoadBalancer Service B is created and the external IP is assigned to it;
antrea-agent processes the creation of Service B first and installs a route for 10.78.144.48 even it already exists, then processes the deletion of Service B and uninstalls the route for 10.78.144.48.

@yuzs2 do you think if the above scenario could happen in your cluster? are the services deleted and created frequently and could the external IPs be reused once it's released? It would be helpful if we have logs about service's creation and deletion somewhere.

alex-vmw · 2022-11-10T18:36:04Z

@tnqn Yes, on DECC Clusters LoadBalancer Services are VERY often created and deleted and of course same IPs are re-used from DNS as long as they are free/released.

tnqn · 2022-11-11T13:39:31Z

@alex-vmw thanks for confirming. Then we could focus on reproducing the issue in this way and fixing it.

github-actions · 2023-02-14T00:39:41Z

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

yuzs2 · 2023-03-14T03:04:38Z

@tnqn any update about the fix? thanks!

tnqn · 2023-03-15T16:36:30Z

@yuzs2 Sorry for the late reply. I have identified the root cause and been working a patch. We will fix it in v1.11.0 and backport it to minor releases up to v1.7.

The issue may be triggered by two cases:

When a LoadBalancer service has multiple ports
When multiple LoadBalancer services share the same LoadBalancerIP

Could you help confirm if the two cases are possible in your cluster? For 2, it depends on LoadBalancer you are using, I see at least AVI supports sharing LoadBalancerIP: https://avinetworks.com/docs/ako/1.7/shared-vip/

alex-vmw · 2023-03-15T17:42:20Z

@tnqn DECC certainly has LoadBalancer services with multiple ports, but DECC doesn't share IPs for the LoadBalancers (each LB will get its own IP). It is possible in DECC that one LB service has the IP first, then gets deleted and a different LB will get the same IP.

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of servicePortNames. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes antrea-io#4361 Signed-off-by: Quan Tian <[email protected]>

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes antrea-io#4361 Signed-off-by: Quan Tian <[email protected]>

tnqn · 2023-03-16T13:54:47Z

Thanks @alex-vmw for confirming. #4711 should fix it. Will let you know when a patch release is available.

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes antrea-io#4361 Signed-off-by: Quan Tian <[email protected]>

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes #4361 Signed-off-by: Quan Tian <[email protected]>

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes antrea-io#4361 Signed-off-by: Quan Tian <[email protected]>

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes #4361 Signed-off-by: Quan Tian <[email protected]>

tnqn · 2023-03-30T14:24:31Z

@yuzs2 @alex-vmw The latest release v1.11.0 has included the fix. It has also been backported to v1.7.3, and will be backported to v1.8.x, v1.9.x, v1.10.x in the coming days.

When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes #4361 Signed-off-by: Quan Tian <[email protected]>

…o#4711) When proxyAll is enabled, AntreaProxy needs to install routes in the host network namespace to redirect traffic to OVS for load balancing. For a Service with multiple ports, multiple ServicePorts are generated and processed. The previous code installed the route for a ClusterIP or a LoadBalancerIP multiple times when such a Service was created, and uninstalled the route multiple times when it was deleted, leading to a few problems. This patch adds a serviceIPRouteReferences which tracks the references of Service IPs' routes. The key is the Service IP and the value is the the set of ServiceInfo strings. With the references, we install a route exactly once as long as it's used by any ServicePorts and uninstall it exactly once when it's no longer used by any ServicePorts. This patch also fixes an issue that the route for ClusterIP was not removed on Windows Nodes after the Service was removed. Fixes antrea-io#4361 Signed-off-by: Quan Tian <[email protected]>

yuzs2 added the kind/bug Categorizes issue or PR as related to a bug. label Nov 1, 2022

hongliangl self-assigned this Nov 1, 2022

tnqn added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Nov 15, 2022

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 14, 2023

hongliangl removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 14, 2023

tnqn mentioned this issue Mar 16, 2023

Fix route deletion for Service ClusterIP and LoadBalancerIP #4711

Merged

tnqn closed this as completed in #4711 Mar 17, 2023

tnqn added the reported-by/end-user Issues reported by end users. label Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

route entry would disappear after the LoadBalancer is registered #4361

route entry would disappear after the LoadBalancer is registered #4361

yuzs2 commented Nov 1, 2022 •

edited

Loading

tnqn commented Nov 8, 2022

hongliangl commented Nov 8, 2022

tnqn commented Nov 8, 2022

yuzs2 commented Nov 9, 2022

tnqn commented Nov 10, 2022

alex-vmw commented Nov 10, 2022

tnqn commented Nov 11, 2022

github-actions bot commented Feb 14, 2023

yuzs2 commented Mar 14, 2023

tnqn commented Mar 15, 2023

alex-vmw commented Mar 15, 2023

tnqn commented Mar 16, 2023

tnqn commented Mar 30, 2023 •

edited

Loading

route entry would disappear after the LoadBalancer is registered #4361

route entry would disappear after the LoadBalancer is registered #4361

Comments

yuzs2 commented Nov 1, 2022 • edited Loading

tnqn commented Nov 8, 2022

hongliangl commented Nov 8, 2022

tnqn commented Nov 8, 2022

yuzs2 commented Nov 9, 2022

tnqn commented Nov 10, 2022

alex-vmw commented Nov 10, 2022

tnqn commented Nov 11, 2022

github-actions bot commented Feb 14, 2023

yuzs2 commented Mar 14, 2023

tnqn commented Mar 15, 2023

alex-vmw commented Mar 15, 2023

tnqn commented Mar 16, 2023

tnqn commented Mar 30, 2023 • edited Loading

yuzs2 commented Nov 1, 2022 •

edited

Loading

tnqn commented Mar 30, 2023 •

edited

Loading