Add support for clusterset IP for MCS Compliance

[vthapar]

Epic Description

While Submariner/Lighthouse implements the MCS API, it is not compliant in that there is no clusterset IP assigned in the ServiceImport.

While there are design reasons mostly to do with perf and scale, there are compelling use cases to have Submariner provide an implementation that is compliant with MCS API in this respect. e.g. Istio with Submariner.

Acceptance Criteria

Submariner deployment is compliant with MCS API wrt clusterset IP assignment.

Any MCS SIG compliance tests that can serve as acceptance tests? Yes

Definition of Done (Checklist)

• Code complete
• Relevant metrics added
• The acceptance criteria met
• Unit/e2e test added & pass
• CI jobs pass
• Deployed using cloud-prepare+subctl
• Deployed using ACM/OCM addon
• Deploy using Helm
• Deployed on supported platforms (for e.g kind, OCP on AWS, OCP on GCP)
• Run subctl verify, diagnose and gather
• Uninstall
• Troubleshooting (gather/diagnose) added
• Documentation added
• Release notes added

Work Items

• Enhancement Proposal with all the changes required for MCS Compliance mode
• Changes to Submariner and/or ServiceDiscovery CRs to add option for MCS mode
• Changes to Lighthouse Agent
• Changes to Lighthouse CoreDNS
• Changes to Submariner-Operator to deploy Lighthouse in MCS Compliance mode
• Changes to Submariner-Operator to create ConfigMap to store the allocated clusterset CIDRs
• Changes to subctl
• Lighthouse E2E test
• Changes to the helm charts

[tpantelis]

2. EndpointSlices do not contain individual Pod IPs but ClusterIP of services in that cluster.

Looking at https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api#tracking-endpoints, I don't see where it explicitly states that EndpointSlices have to contain individual Pod IPs. But perhaps I'm missing something.

Also it's optional to even use EndpointSlices.

BTW, I'm not saying we can't/shouldn't expose the Pod IPs - just pointing out that I don't think we're non-compliant.

[tpantelis]

For 1), a prior issue was reported but went stale.

For 2), it was previously discussed here.

[vthapar]

The spec mentions this:

One or more EndpointSlice resources will exist for the exported Service, with each EndpointSlice containing only endpoints from a single source cluster.

Here it does say that EndpointSlice should contain Endpoints from that cluster. ES are optional for an implementation but spec mentions that if created they should contain endpoints, which basically means pod ips.

Should we reword the Epic? It is more a use case from Istio, but I wanted to use a generic wording.

[tpantelis]

Here it does say that EndpointSlice should contain Endpoints from that cluster. ES are optional for an implementation but spec mentions that if created they should contain endpoints, which basically means pod ips.

I guess it's a matter of semantics. Generically an "endpoint is an API object that acts as a bridge between a service and the pods that fulfill that service's requests". However I don't think that necessarily means it has to be an actual pod IP - in our case it's the service VIP that indirectly leads to the backing pods.

I think the key point in that statement is that the endpoints contained in an EPS, whatever they are/represent, must all be from the same cluster.

Should we reword the Epic? It is more a use case from Istio, but I wanted to use a generic wording.

We can mention the use case is for Istio (which we just did :)).