Support shared LoadBalancerIP for multiple Services

Users want to share LoadBalancerIP between multiple Services when they
face external IP shortage. It's possible to do it when the Services
sharing an IP meet the requirements:

* The Services use different ports
* The Services use the `Cluster` external traffic policy, or they have
  identical Endpoints.

However, the ability of using any IP that is already allocated to another
Service may incur a security risk that a Service can "steal" LoadBalancer
traffic intended for another Service.

To support the use case without introducing the security risk, we use
the annotation `service.antrea.io/allow-shared-load-balancer-ip: true`
on Services to restrict IPs that can be shared. Services without the
annotation will continue to have their LoadBalancerIPs exclusively used.
Services with the annotation can share an IP between themselves when
requesting the same IP.

Ideally, we should also check if the Services meet the first two
requirements before assigning the IP to them. However, it's difficult to
prevent Services from being changed to not meet the requirements after
they get the IP assigned. Therefore, we assume that users using the
feature know how to configure Services properly.

Signed-off-by: Quan Tian <[email protected]>

docs/service-loadbalancer.md

@@ -165,6 +165,69 @@ spec:
   type: LoadBalancer
 ```
 
+By default, Antrea doesn't allocate a single IP to multiple Services. Before
+Antrea v2.1, if multiple Services requested the same IP, only one of them would
+get the IP assigned. Starting with Antrea v2.1, to share an IP between multiple
+Services, you can annotate the Services with
+`service.antrea.io/allow-shared-load-balancer-ip: true` when requesting a
+particular IP. Note that the IP will only be shared between Services having the
+annotation. If not all Services are annotated, the IP may either be allocated
+to one of the unannotated Services or shared between the annotated Services,
+depending on the order in which they are processed. The annotation only takes
+effect during the IP allocation phase. Once the IP has been allocated, removing
+this annotation from a Service will not result in the IP being reclaimed from
+it or other Services.
+
+For example, the following two Services will share an IP:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-service-1
+  annotations:
+    service.antrea.io/external-ip-pool: "service-external-ip-pool"
+    service.antrea.io/allow-shared-load-balancer-ip: "true"
+spec:
+  selector:
+    app: MyApp1
+  loadBalancerIP: "10.10.0.2"
+  ports:
+    - protocol: TCP
+      port: 80
+      targetPort: 80
+  type: LoadBalancer
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-service-2
+  annotations:
+    service.antrea.io/external-ip-pool: "service-external-ip-pool"
+    service.antrea.io/allow-shared-load-balancer-ip: "true"
+spec:
+  selector:
+    app: MyApp2
+  loadBalancerIP: "10.10.0.2"
+  ports:
+    - protocol: TCP
+      port: 8080
+      targetPort: 8080
+  type: LoadBalancer
+```
+
+Note that sharing a LoadBalancer IP between multiple Services only works under
+the following conditions:
+
+* The Services use different ports.
+* The Services use the `Cluster` external traffic policy. Sharing a
+  LoadBalancer IP between Services using the `Local` external traffic policy
+  can also work if they have identical Endpoints. However, in such cases, it
+  may be preferable to consolidate the Services into a single Service.
+
+Otherwise, the datapath may not work even though the IP is allocated to the
+Services successfully.
+
 #### Validate Service external IP
 
 Once Antrea allocates an external IP for a Service of type LoadBalancer, it

pkg/agent/types/annotations.go

@@ -30,6 +30,9 @@ const (
 	// ServiceExternalIPPoolAnnotationKey is the key of the Service annotation that specifies the Service's desired external IP pool.
 	ServiceExternalIPPoolAnnotationKey string = "service.antrea.io/external-ip-pool"
 
+	// ServiceAllowSharedIPAnnotationKey is the key of the Service annotation that specifies whether the Service is allowed to use a shared LoadBalancer IP.
+	ServiceAllowSharedIPAnnotationKey string = "service.antrea.io/allow-shared-load-balancer-ip"
+
 	// ServiceLoadBalancerModeAnnotationKey is the key of the Service annotation that specifies the Service's load balancer mode.
 	ServiceLoadBalancerModeAnnotationKey string = "service.antrea.io/load-balancer-mode"