Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Egress HA assigns different egress ips to same egress interface on same egress node #6836

Open
rajnkamr opened this issue Nov 28, 2024 · 1 comment
Labels
area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). kind/bug Categorizes issue or PR as related to a bug.

Comments

@rajnkamr
Copy link
Contributor

Describe the bug

Egress HA assigns different egress ips to same egress interface on same egress node. It should be avoided unless nodeselector is provided as same egress node.
NAME EGRESSIP AGE NODE
egress-prod-web 172.18.0.11 19h bgp-worker2

egress-staging-web 172.18.0.12 19h bgp-worker2

Egress Interface

14: antrea-egress0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether e2:12:06:f2:dc:2f brd ff:ff:ff:ff:ff:ff inet 172.18.0.11/32 scope global antrea-egress0 valid_lft forever preferred_lft forever

inet 172.18.0.12/32 scope global antrea-egress0 valid_lft forever preferred_lft forever

To Reproduce

Apply below config
1.externalippool.yaml

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
  name: external-ip-pool
spec:
  ipRanges:
  - start: 172.18.0.11  # 172.18.0.11-172.18.0.20 can be used as Egress IPs
    end: 172.18.0.20
  nodeSelector: {}     # All Nodes can be Egress Nodes`

2.egress1.yaml

apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-prod-web
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: prod
    podSelector:
      matchLabels:
        app: web
  externalIPPool: external-ip-pool`

3.egress2.yaml

apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-staging-web
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: staging
    podSelector:
      matchLabels:
        app: web
  externalIPPool: external-ip-pool`

Expected

Unless nodeselector is a single egress node, the expectation should be to assign egress ip to different interfaces on different nodes to avoid external traffic disruption for all egress workloads !

Actual behavior

Versions:

Additional context

@rajnkamr rajnkamr added kind/bug Categorizes issue or PR as related to a bug. area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). labels Nov 28, 2024
@jainpulkit22
Copy link
Contributor

jainpulkit22 commented Nov 29, 2024

There is already a concept of maxEgressIPs per node, so if the egressIPs per node is less that threshold we can still assign more IPs to the egress interface on that node without having any traffic disruptions. And selecting a node for egress is a random process and it can select any node if that node has not reach the threshold for max number of EgressIPs, and I think that this behaviour is correct also because we cannot restrict a node to have only one EgressIP.
So, IMO this is expected behaviour and is correct also.

cc: @antoninbas @tnqn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants