Add spec for multi-network scenario

keps/sig-multicluster/1645-multi-cluster-services-api/README.md

@@ -260,6 +260,12 @@ nitty-gritty.
   The cluster name should be consistent for the life of a cluster and its
   membership in the clusterset. Implementations should treat name mutation as
   a delete of the membership followed by recreation with the new name.
+- **cluster network** - An identifier for the cluster network. Each cluster can have an optional name that can identify the network its running in. The network name must be a valid [RFC
+  1123](https://tools.ietf.org/html/rfc1123) DNS label. Two or more clusters within the ClusterSet can have the same network identifier.
+
+  The network name should be consistent during its
+  membership in the clusterset. Implementations should treat network change as
+  a delete of the membership followed by recreation with the new name.
 
 [namespace sameness]: https://github.com/kubernetes/community/blob/master/sig-multicluster/namespace-sameness-position-statement.md
 
@@ -664,6 +670,58 @@ endpoints:
 The `ServiceImport.Spec.IP` (VIP) can be used to access this service from within
 this cluster.
 
+
+#### Multi-network scenario
+One of more clusters in a ClusterSet can be running on a discrete network (a non-flat network). An MCS controller can use the `network.k8s.io` `ClusterProperty` to determine if a cluster in a `ClusterSet` is running on a discrete network. In this scenario, the generated EndpointSlice in an importing cluster will point to the FQDN (Fully Qualified Domain Name) of the Gateway/Loadbalancer of the cluster this `EndpointSlice` belongs to. The MCS controller implementation is responsible for determining the Gateway/Loadbalancer in a cluster for a non-flat network. A sample implementation can use a K8s [Gateway](https://gateway-api.sigs.k8s.io/) defined with an annotation `network.k8s.io: mcs-gateway` to match against a ServiceExport with the same annotation to identity the FQDN associated with it. This FQDN would then be used in the EndpointSlice generated for this cluster in the importing clusters. The MCS controller may implement creating the K8s Gateway and the necessary routing rules to automate the configuration.
+
+```yaml
+apiVersion: multicluster.k8s.io/v1alpha1
+kind: ServiceImport
+metadata:
+  name: my-svc
+  namespace: my-ns
+spec:
+  ips:
+  - 42.42.42.42
+  type: "ClusterSetIP"
+  ports:
+  - name: http
+    protocol: TCP
+    port: 80
+  sessionAffinity: None
+status:
+  clusters:
+  - cluster: us-west2-a-my-cluster
+---
+apiVersion: discovery.k8s.io/v1beta1
+kind: EndpointSlice
+metadata:
+  name: imported-my-svc-cluster-b-1
+  namespace: my-ns
+  labels:
+    multicluster.kubernetes.io/source-cluster: us-west2-a-my-cluster
+    multicluster.kubernetes.io/service-name: my-svc
+  ownerReferences:
+  - apiVersion: multicluster.k8s.io/v1alpha1
+    controller: false
+    kind: ServiceImport
+    name: my-svc
+addressType: FQDN
+ports:
+  - name: http
+    protocol: TCP
+    port: 80
+endpoints:
+  hostname: us-west2-a-my-cluster.subdomain.domain.com
+    conditions:
+      ready: true
+    topology:
+     topology.kubernetes.io/zone: us-west2-a
+```
+
+##### Known limitation
+In a multi-network scenario where the `EndpointSlice`s are backed by FQDNs, there isn't currently a way (K8s native support) to proportionately distribute the traffic based on the number of endpoints behind these FQDNs. There is active ongoing work in SIG-Network to add an attribute to represent the number of endpoints for `EndpointSlice`s when the addressType is FQDN. This will provide a way for kube-proxy to load balance across `EndpointSlice`s of addressType FQDN.
+
 ### ClusterSet Service Behavior Expectations
 
 #### Service Types

keps/sig-multicluster/2149-clusterid/README.md

@@ -294,6 +294,9 @@ My controller interacts with multiple clusters and needs to disambiguate between
 
 _For example, [CAPN's virtualcluster project](https://github.com/kubernetes-sigs/cluster-api-provider-nested) is implementing a multi-tenant scheduler that schedules tenant namespaces only in certain parent clusters, and a separate syncer running in each parent cluster controller needs to compare the name of the parent cluster to determine whether the namespace should be synced. ([ref](https://github.com/kubernetes/enhancements/issues/2149#issuecomment-768486457))._
 
+#### Multi-network scenario
+
+With in a ClusterSet I have one or more clusters where pods across these clusters are not directly routable (a non-flat network).
 
 ### `ClusterProperty` CRD
 
@@ -353,6 +356,38 @@ Contains a unique identifier for the containing cluster.
 
 **Reusing cluster names**: Since an `id.k8s.io ClusterProperty` has no restrictions on whether or not a ClusterProperty can be repeatable, if a cluster unregisters from a ClusterSet it is permitted under this standard to rejoin later with the same `id.k8s.io ClusterProperty` it had before. Similarly, a *different* cluster could join a ClusterSet with the same `id.k8s.io ClusterProperty` that had been used by another cluster previously, as long as both do not have membership in the same ClusterSet at the same time. Finally, two or more clusters may have the same `id.k8s.io ClusterProperty` concurrently (though they **should** not; see "Uniqueness" above) *as long as* they both do not have membership in the same ClusterSet.
 
+#### Property: `network.k8s.io`
+
+Contains an identifier representing the network for the cluster.
+
+
+##### Uniqueness
+
+*   The identifier **need not be** unique but **should** remain the same for a cluster for the duration of the cluster’s membership.
+
+##### Lifespan
+
+*   The identifier if exists **should** be immutable for the lifespan of a ClusterSet membership.
+
+
+##### Contents
+
+*   The identifier **must** be a valid RFC-1123 DNS label [as described for object names in the Kubernetes docs](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names).
+    *   Following the most restrictive standard naming constraint ensures maximum usefulness and portability.
+*   The identifier **may** be a human readable description.
+
+
+##### Consumers
+
+*   **Must** be able to rely on the identifier, if exists, unmodified for the entire duration of its membership in a ClusterSet.
+*   **Should** watch the `network.k8s.io` property to handle potential changes if they live beyond the ClusterSet membership.
+*   **May** rely on the existence of an identifier for clusters that do not belong to a ClusterSet so long as the implementation provides one.
+
+
+##### Notable scenarios
+
+**Cluster changes its network**: Since a `network.k8s.io ClusterProperty` must be immutable for the duration of its *membership* in a given ClusterSet, the property contents can be "changed" by unregistering the cluster from the ClusterSet and reregistering it with the new name.
+
 #### Property: `clusterset.k8s.io`
 
 Contains an identifier that relates the containing cluster to the ClusterSet in which it belongs.