ExternalTrafficPolicy: local not working well

[manuelbuil]

Environmental Info:
K3s Version:

Any

Node(s) CPU architecture, OS, and Version:

Any

Cluster Configuration:

2 nodes. For example: 1 server, 1 agent

Describe the bug:

When deploying a pod with an HTTP server and then a service of type LoadBalancer that uses that HTTP server, you will get 2 external IPs, which are the IPs of the nodes. If you change the config of the service to ExternalTrafficPolicy: local, the number of externalIPs is reduced to 1, which is the IP of the node where the pod is sitting. However, if you query the service using the other node IP, you'll still be able to reach it.

The reason is that svclb daemon is still running a pod in the node which does not have the HTTP server pod.

Steps To Reproduce:

• Installed K3s:

Explained above

Expected behavior:

Traffic can't reach the node where the HTTP server pod is running when ExternalTrafficPolicy: local

Actual behavior:

Traffic can reach the node where the HTTP server pod is running when ExternalTrafficPolicy: local

Additional context / logs:

[fmoral2]

validate revert here
#11078
#11080
#11081
#11082