From 96404fdd79b20eab557f8abe3a5496ef22923bc3 Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Tue, 27 Oct 2026 10:40:11 -0700 Subject: [PATCH 1/9] feat(operator): export Global Router pools through the Multi-Cluster Services API With multiCluster.mcsImplementation set to mcs or karmada, the operator creates a multicluster.x-k8s.io/v1alpha1 ServiceExport for the -pool-relay and -frontend Services of each DGD annotated nvidia.com/global-router-pool: "true". The DGD owns the exports, and removing the annotation withdraws them. Karmada reads the same ServiceExport in member clusters; with karmada the operator skips a Service name longer than 55 characters, which Karmada cannot import. A new feature gate, multiClusterExport, verifies the ServiceExport API at startup. Each workload program reconciles the exports after its workloads, and a watch on ServiceExports requeues the DGDs that export a Service of that name. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- deploy/helm/charts/platform/README.md | 1 + .../components/operator/files/role.yaml | 10 + .../operator/templates/operator-config.yaml | 5 + .../tests/multi_cluster_config_test.yaml | 20 ++ deploy/helm/charts/platform/values.yaml | 12 + deploy/operator/api/config/v1alpha1/types.go | 30 ++ .../config/v1alpha1/zz_generated.deepcopy.go | 16 ++ .../api/config/validation/validation.go | 13 + .../api/config/validation/validation_test.go | 23 ++ deploy/operator/internal/consts/consts.go | 5 + .../controller/dgd_component_program.go | 9 + .../dgd_disaggregatedset_program.go | 9 + .../internal/controller/dgd_grove_program.go | 9 + .../dgd_service_export_reconciler.go | 180 ++++++++++++ .../dgd_service_export_reconciler_test.go | 272 ++++++++++++++++++ .../dynamographdeployment_controller.go | 16 ++ deploy/operator/internal/features/gates.go | 30 +- .../operator/internal/features/gates_test.go | 21 +- .../internal/features/multicluster.go | 38 +++ .../internal/features/multicluster_test.go | 69 +++++ .../internal/multicluster/serviceexport.go | 92 ++++++ .../multicluster/serviceexport_test.go | 82 ++++++ .../additional-resources/api-reference-k8s.md | 39 +++ .../dynamo-graph-deployment.mdx | 6 + .../kubernetes-api/full-api-reference.mdx | 41 ++- 25 files changed, 1026 insertions(+), 22 deletions(-) create mode 100644 deploy/helm/charts/platform/tests/multi_cluster_config_test.yaml create mode 100644 deploy/operator/internal/controller/dgd_service_export_reconciler.go create mode 100644 deploy/operator/internal/controller/dgd_service_export_reconciler_test.go create mode 100644 deploy/operator/internal/features/multicluster.go create mode 100644 deploy/operator/internal/features/multicluster_test.go create mode 100644 deploy/operator/internal/multicluster/serviceexport.go create mode 100644 deploy/operator/internal/multicluster/serviceexport_test.go diff --git a/deploy/helm/charts/platform/README.md b/deploy/helm/charts/platform/README.md index e8ec469a67ae..569ece08b7fa 100644 --- a/deploy/helm/charts/platform/README.md +++ b/deploy/helm/charts/platform/README.md @@ -574,6 +574,7 @@ Kubernetes: `>=1.30.0-0` | dynamo-operator.dynamo.serviceMesh.istio.clientCertificate | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS certificate used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.privateKey | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS private key used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.caCertificates | string | `""` | Optional path (in the istio-proxy sidecar's filesystem) to CA certificates used to verify the EPP server certificate. Used only when tlsMode is "MUTUAL". | +| dynamo-operator.dynamo.multiCluster.mcsImplementation | string | `""` | MCS implementation that imports the exports: "mcs" for the SIG Multicluster API or "karmada". For each DGD annotated `nvidia.com/global-router-pool: "true"`, the operator then creates a ServiceExport for `-pool-relay` and `-frontend`. Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this cluster. Empty exports nothing; clearing it leaves earlier exports in place until their DGDs are deleted. | | dynamo-operator.dynamo.metrics.prometheusEndpoint | string | `""` | Endpoint that services can use to retrieve metrics. If set, dynamo operator will automatically inject the PROMETHEUS_ENDPOINT environment variable into services it manages. Users can override the value of the PROMETHEUS_ENDPOINT environment variable by modifying the corresponding deployment's environment variables | | dynamo-operator.dynamo.mpiRun.secretName | string | `"mpi-run-ssh-secret"` | Name of the secret containing the SSH key for MPI Run | | dynamo-operator.webhook.certificateSecret.name | string | `"webhook-server-cert"` | Name of the Kubernetes secret containing webhook TLS certificates. The secret must contain three keys: tls.crt (server certificate), tls.key (server private key), and ca.crt (Certificate Authority certificate). | diff --git a/deploy/helm/charts/platform/components/operator/files/role.yaml b/deploy/helm/charts/platform/components/operator/files/role.yaml index 49e0d5309ea6..9b23a11ff9ee 100644 --- a/deploy/helm/charts/platform/components/operator/files/role.yaml +++ b/deploy/helm/charts/platform/components/operator/files/role.yaml @@ -238,6 +238,16 @@ rules: - patch - update - watch +- apiGroups: + - multicluster.x-k8s.io + resources: + - serviceexports + verbs: + - create + - delete + - get + - list + - watch - apiGroups: - networking.istio.io resources: diff --git a/deploy/helm/charts/platform/components/operator/templates/operator-config.yaml b/deploy/helm/charts/platform/components/operator/templates/operator-config.yaml index fd8470b34862..b3cd0ad06797 100644 --- a/deploy/helm/charts/platform/components/operator/templates/operator-config.yaml +++ b/deploy/helm/charts/platform/components/operator/templates/operator-config.yaml @@ -182,6 +182,11 @@ data: {{- end }} {{- end }} {{- end }} + {{- $mcsImplementation := (.Values.dynamo.multiCluster | default dict).mcsImplementation | default "" }} + {{- if $mcsImplementation }} + multiCluster: + mcsImplementation: {{ $mcsImplementation | quote }} + {{- end }} {{- if not .Values.namespaceRestriction.enabled }} rbac: plannerClusterRoleName: {{ include "dynamo-operator.fullname" . }}-planner diff --git a/deploy/helm/charts/platform/tests/multi_cluster_config_test.yaml b/deploy/helm/charts/platform/tests/multi_cluster_config_test.yaml new file mode 100644 index 000000000000..afa6d286a15f --- /dev/null +++ b/deploy/helm/charts/platform/tests/multi_cluster_config_test.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: multi-cluster operator configuration +templates: + - charts/dynamo-operator/templates/operator-config.yaml +tests: + - it: exports nothing by default + asserts: + - notMatchRegex: + path: data["config.yaml"] + pattern: "multiCluster:" + + - it: passes the MCS implementation to the operator + set: + dynamo-operator.dynamo.multiCluster.mcsImplementation: karmada + asserts: + - matchRegex: + path: data["config.yaml"] + pattern: 'multiCluster:\n\s+mcsImplementation: "karmada"' diff --git a/deploy/helm/charts/platform/values.yaml b/deploy/helm/charts/platform/values.yaml index 13c14c9c058f..7f76f23892ac 100644 --- a/deploy/helm/charts/platform/values.yaml +++ b/deploy/helm/charts/platform/values.yaml @@ -246,6 +246,18 @@ dynamo-operator: # when tlsMode is "MUTUAL". caCertificates: "" + # Exports of Global Router pools through the Kubernetes Multi-Cluster + # Services (MCS) API, for the Global Discovery Service's Kubernetes stores. + multiCluster: + # -- MCS implementation that imports the exports: "mcs" for the SIG + # Multicluster API or "karmada". For each DGD annotated + # `nvidia.com/global-router-pool: "true"`, the operator then creates a + # ServiceExport for `-pool-relay` and `-frontend`. + # Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this + # cluster. Empty exports nothing; clearing it leaves earlier exports in + # place until their DGDs are deleted. + mcsImplementation: "" + # Metrics configuration metrics: # -- Endpoint that services can use to retrieve metrics. If set, dynamo operator will automatically inject the PROMETHEUS_ENDPOINT environment variable into services it manages. Users can override the value of the PROMETHEUS_ENDPOINT environment variable by modifying the corresponding deployment's environment variables diff --git a/deploy/operator/api/config/v1alpha1/types.go b/deploy/operator/api/config/v1alpha1/types.go index 01d310e3016d..76a92bf00153 100644 --- a/deploy/operator/api/config/v1alpha1/types.go +++ b/deploy/operator/api/config/v1alpha1/types.go @@ -62,6 +62,9 @@ type OperatorConfiguration struct { // Discovery backend configuration Discovery DiscoveryConfiguration `json:"discovery"` + // MultiCluster configures exporting Global Router pools to other clusters. + MultiCluster MultiClusterConfiguration `json:"multiCluster,omitempty"` + // GPU discovery configuration GPU GPUConfiguration `json:"gpu"` @@ -397,6 +400,33 @@ type DiscoveryConfiguration struct { Backend DiscoveryBackend `json:"backend"` } +// MultiClusterConfiguration configures how the operator exports Global Router +// pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global +// Discovery Service's Kubernetes stores. The operator exports a pool only when +// MCSImplementation is set and its DGD has the nvidia.com/global-router-pool +// annotation. +type MultiClusterConfiguration struct { + // MCSImplementation is the MCS implementation that imports the exports. Empty, + // the default, exports nothing: Global Router pools are then discovered without + // Kubernetes. Clearing it leaves earlier exports in place until their DGDs are + // deleted. + MCSImplementation MCSImplementation `json:"mcsImplementation,omitempty"` +} + +// MCSImplementation identifies an implementation of the Multi-Cluster Services API. +type MCSImplementation string + +const ( + // MCSImplementationNone exports nothing. + MCSImplementationNone MCSImplementation = "" + // MCSImplementationStandard is an implementation of the SIG Multicluster API that + // imports Services into ServiceImports and EndpointSlices with the standard labels. + MCSImplementationStandard MCSImplementation = "mcs" + // MCSImplementationKarmada is Karmada, which reads the same ServiceExport in + // member clusters but names imported EndpointSlices after the source cluster. + MCSImplementationKarmada MCSImplementation = "karmada" +) + // DiscoveryBackend is the type for the discovery backend. type DiscoveryBackend string diff --git a/deploy/operator/api/config/v1alpha1/zz_generated.deepcopy.go b/deploy/operator/api/config/v1alpha1/zz_generated.deepcopy.go index 952d37192b26..a72edb97db00 100644 --- a/deploy/operator/api/config/v1alpha1/zz_generated.deepcopy.go +++ b/deploy/operator/api/config/v1alpha1/zz_generated.deepcopy.go @@ -323,6 +323,21 @@ func (in *MetricsServer) DeepCopy() *MetricsServer { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *MultiClusterConfiguration) DeepCopyInto(out *MultiClusterConfiguration) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MultiClusterConfiguration. +func (in *MultiClusterConfiguration) DeepCopy() *MultiClusterConfiguration { + if in == nil { + return nil + } + out := new(MultiClusterConfiguration) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *NamespaceConfiguration) DeepCopyInto(out *NamespaceConfiguration) { *out = *in @@ -372,6 +387,7 @@ func (in *OperatorConfiguration) DeepCopyInto(out *OperatorConfiguration) { out.MPI = in.MPI in.Checkpoint.DeepCopyInto(&out.Checkpoint) out.Discovery = in.Discovery + out.MultiCluster = in.MultiCluster in.GPU.DeepCopyInto(&out.GPU) out.Logging = in.Logging out.Security = in.Security diff --git a/deploy/operator/api/config/validation/validation.go b/deploy/operator/api/config/validation/validation.go index e251edd690bb..4f3cd8a44bdb 100644 --- a/deploy/operator/api/config/validation/validation.go +++ b/deploy/operator/api/config/validation/validation.go @@ -39,6 +39,7 @@ func ValidateOperatorConfiguration(config *configv1alpha1.OperatorConfiguration) validateMPI(&config.MPI, field.NewPath("mpi")), validateInfrastructure(&config.Infrastructure, field.NewPath("infrastructure")), validateDiscovery(&config.Discovery, field.NewPath("discovery")), + validateMultiCluster(&config.MultiCluster, field.NewPath("multiCluster")), validateRBAC(config), validateOrchestrators(&config.Orchestrators, field.NewPath("orchestrators")), validateIngress(&config.Ingress, field.NewPath("ingress")), @@ -209,6 +210,18 @@ func validateDiscovery(discovery *configv1alpha1.DiscoveryConfiguration, fldPath return allErrs } +func validateMultiCluster(multiCluster *configv1alpha1.MultiClusterConfiguration, fldPath *field.Path) field.ErrorList { + switch multiCluster.MCSImplementation { + case configv1alpha1.MCSImplementationNone, configv1alpha1.MCSImplementationStandard, configv1alpha1.MCSImplementationKarmada: + return nil + } + return field.ErrorList{field.NotSupported( + fldPath.Child("mcsImplementation"), + multiCluster.MCSImplementation, + []string{"", "mcs", "karmada"}, + )} +} + // validateRBAC is mode-aware: validates RBAC fields based on namespace mode. func validateRBAC(config *configv1alpha1.OperatorConfiguration) field.ErrorList { allErrs := field.ErrorList{} diff --git a/deploy/operator/api/config/validation/validation_test.go b/deploy/operator/api/config/validation/validation_test.go index e713c8a68dc3..f9d75025e9d3 100644 --- a/deploy/operator/api/config/validation/validation_test.go +++ b/deploy/operator/api/config/validation/validation_test.go @@ -86,6 +86,29 @@ func TestValidateOperatorConfiguration_InvalidDiscoveryBackend(t *testing.T) { } } +func TestValidateOperatorConfiguration_MCSImplementation(t *testing.T) { + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + wantErrors int + }{ + {name: "none", implementation: configv1alpha1.MCSImplementationNone}, + {name: "standard MCS", implementation: configv1alpha1.MCSImplementationStandard}, + {name: "Karmada", implementation: configv1alpha1.MCSImplementationKarmada}, + {name: "unknown", implementation: "gke", wantErrors: 1}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Log("Validate a configuration that exports Global Router pools through this implementation") + cfg := validConfig() + cfg.MultiCluster.MCSImplementation = tc.implementation + errs := ValidateOperatorConfiguration(cfg) + if len(errs) != tc.wantErrors { + t.Errorf("expected %d errors, got %d: %v", tc.wantErrors, len(errs), errs) + } + }) + } +} + func TestValidateOperatorConfiguration_ClusterWideMissingPlannerRole(t *testing.T) { cfg := validConfig() cfg.RBAC.PlannerClusterRoleName = "" diff --git a/deploy/operator/internal/consts/consts.go b/deploy/operator/internal/consts/consts.go index f926af4a9db7..732fc7b1af20 100644 --- a/deploy/operator/internal/consts/consts.go +++ b/deploy/operator/internal/consts/consts.go @@ -61,6 +61,11 @@ const ( // when the namespace carries istio-injection=enabled. KubeAnnotationIstioSidecarInject = "sidecar.istio.io/inject" + // KubeAnnotationGlobalRouterPool set to "true" marks a DGD as a Global Router + // pool. When the operator is configured with a Multi-Cluster Services + // implementation, it exports the pool's PoolRelay and frontend Services. + KubeAnnotationGlobalRouterPool = "nvidia.com/global-router-pool" + KubeAnnotationDisableImagePullSecretDiscovery = "nvidia.com/disable-image-pull-secret-discovery" KubeAnnotationDynamoDiscoveryBackend = "nvidia.com/dynamo-discovery-backend" KubeAnnotationDynamoKubeDiscoveryMode = "nvidia.com/dynamo-kube-discovery-mode" diff --git a/deploy/operator/internal/controller/dgd_component_program.go b/deploy/operator/internal/controller/dgd_component_program.go index 6de7c12faf45..a5df983f0730 100644 --- a/deploy/operator/internal/controller/dgd_component_program.go +++ b/deploy/operator/internal/controller/dgd_component_program.go @@ -35,6 +35,7 @@ type componentProgram struct { restartProgress *componentRestartProgressResolver workloads *componentWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler + serviceExports *dgdServiceExportReconciler lwsEnabled bool } @@ -56,6 +57,7 @@ func (r *DynamoGraphDeploymentReconciler) newComponentProgram() *componentProgra restartProgress: newComponentRestartProgressResolver(r.Client), workloads: newComponentWorkloadsReconciler(r.Client, r.Recorder, rollout), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), + serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), lwsEnabled: r.RuntimeConfig.Gate.Enabled(features.LWS), } } @@ -157,6 +159,13 @@ func (p *componentProgram) Reconcile( return programResult, fmt.Errorf("failed to reconcile scaling adapters: %w", err) } } + + // A Global Router pool's exports follow its workloads, so a failed export does + // not hold them back. + if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { + log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") + return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) + } programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_disaggregatedset_program.go b/deploy/operator/internal/controller/dgd_disaggregatedset_program.go index e3b58d38c168..8b44d378b13e 100644 --- a/deploy/operator/internal/controller/dgd_disaggregatedset_program.go +++ b/deploy/operator/internal/controller/dgd_disaggregatedset_program.go @@ -39,6 +39,7 @@ type disaggregatedSetProgram struct { rollout *dgdWorkerRolloutReconciler workloads *disaggregatedSetWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler + serviceExports *dgdServiceExportReconciler gate features.Gate } @@ -57,6 +58,7 @@ func (r *DynamoGraphDeploymentReconciler) newDisaggregatedSetProgram() *disaggre rollout: rollout, workloads: r.newDisaggregatedSetWorkloadsReconciler(rollout), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), + serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), gate: r.RuntimeConfig.Gate, } } @@ -139,6 +141,13 @@ func (p *disaggregatedSetProgram) Reconcile( } } + // A Global Router pool's exports follow its workloads, so a failed export does + // not hold them back. + if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { + log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") + return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) + } + programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_grove_program.go b/deploy/operator/internal/controller/dgd_grove_program.go index 4e560cac60ba..2d5cd59a0a26 100644 --- a/deploy/operator/internal/controller/dgd_grove_program.go +++ b/deploy/operator/internal/controller/dgd_grove_program.go @@ -40,6 +40,7 @@ type groveProgram struct { lpxRestartProgress *lpxRestartProgressResolver workloads *groveWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler + serviceExports *dgdServiceExportReconciler topology *dgdGroveTopologyConditionReconciler gate features.Gate lpx *dgdLPXHandoff @@ -106,6 +107,7 @@ func (r *DynamoGraphDeploymentReconciler) newGroveProgram() *groveProgram { r.DockerSecretRetriever, ), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), + serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), topology: newDGDGroveTopologyConditionReconciler(r.Client), gate: r.RuntimeConfig.Gate, lpx: &dgdLPXHandoff{client: r.Client}, @@ -245,6 +247,13 @@ func (p *groveProgram) Reconcile( return programResult, fmt.Errorf("failed to reconcile scaling adapters: %w", err) } } + + // A Global Router pool's exports follow its workloads, so a failed export does + // not hold them back. + if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { + log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") + return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) + } programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler.go b/deploy/operator/internal/controller/dgd_service_export_reconciler.go new file mode 100644 index 000000000000..664d45bec41b --- /dev/null +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler.go @@ -0,0 +1,180 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package controller + +import ( + "context" + "fmt" + "slices" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + nvidiacomv1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/log" +) + +// dgdServiceExportReconciler exports the PoolRelay and frontend Services of a DGD +// marked as a Global Router pool through the configured Multi-Cluster Services +// implementation, and withdraws the exports when the mark is removed. It creates +// and deletes only ServiceExports that the DGD controls. +type dgdServiceExportReconciler struct { + dgdResourceSyncer + config *configv1alpha1.OperatorConfiguration + gate features.Gate +} + +func newDGDServiceExportReconciler( + syncer dgdResourceSyncer, + config *configv1alpha1.OperatorConfiguration, + gate features.Gate, +) *dgdServiceExportReconciler { + return &dgdServiceExportReconciler{dgdResourceSyncer: syncer, config: config, gate: gate} +} + +// Reconcile converges the DGD's ServiceExports on its pool mark. Without a +// configured MCS implementation the operator exports nothing and leaves existing +// exports alone. +func (r *dgdServiceExportReconciler) Reconcile( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, +) error { + if !r.gate.Enabled(features.MultiClusterExport) { + return nil + } + + // Export each of the pool's Services, or withdraw the export. + logger := log.FromContext(ctx) + implementation := r.config.MultiCluster.MCSImplementation + pool := multicluster.IsPool(dgd.Annotations) + maxLength := multicluster.MaxServiceNameLength(implementation) + for _, name := range multicluster.ServiceNames(dgd.Name) { + // A Service the implementation cannot import with its cluster is not exported. + exportable := len(name) <= maxLength + if pool && !exportable { + logger.Error(nil, "Service name is too long to export through this MCS implementation; shorten the DGD name", + "service", name, "implementation", implementation, "maxLength", maxLength) + } + + var err error + if pool && exportable { + err = r.export(ctx, dgd, name) + } else { + err = r.withdraw(ctx, dgd, name) + } + if err != nil { + return err + } + } + return nil +} + +// export creates the ServiceExport for the Service named name unless one exists. +// An existing ServiceExport is left as it is, including one a user created. +func (r *dgdServiceExportReconciler) export( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + name string, +) error { + // An existing export, the DGD's or a user's, needs nothing. + err := r.Get(ctx, types.NamespacedName{Namespace: dgd.Namespace, Name: name}, multicluster.ServiceExportMetadata()) + if err == nil { + return nil + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to get ServiceExport %s: %w", name, err) + } + + // Owned by the DGD, the export is deleted with it. + serviceExport := multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name) + if err := ctrl.SetControllerReference(dgd, serviceExport, r.Scheme()); err != nil { + return fmt.Errorf("failed to set the owner of ServiceExport %s: %w", name, err) + } + if err := r.Create(ctx, serviceExport); err != nil { + // The cache has not seen an export created moments ago. Its watch event + // requeues the DGD. + if apierrors.IsAlreadyExists(err) { + return nil + } + return fmt.Errorf("failed to create ServiceExport %s: %w", name, err) + } + r.recorder.Eventf(dgd, nil, corev1.EventTypeNormal, "CreateServiceExport", "Create", + "Exported Service %s through MCS implementation %s", name, r.config.MultiCluster.MCSImplementation) + return nil +} + +// withdraw deletes the ServiceExport for the Service named name if the DGD +// controls it and it is not already being deleted. +func (r *dgdServiceExportReconciler) withdraw( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + name string, +) error { + // Only an export the DGD controls is withdrawn. + existing := multicluster.ServiceExportMetadata() + err := r.Get(ctx, types.NamespacedName{Namespace: dgd.Namespace, Name: name}, existing) + if apierrors.IsNotFound(err) { + return nil + } + if err != nil { + return fmt.Errorf("failed to get ServiceExport %s: %w", name, err) + } + if !metav1.IsControlledBy(existing, dgd) || existing.DeletionTimestamp != nil { + return nil + } + + // The UID precondition deletes exactly the export just read. + if err := r.Delete(ctx, existing, client.Preconditions{UID: &existing.UID}); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + return fmt.Errorf("failed to delete ServiceExport %s: %w", name, err) + } + r.recorder.Eventf(dgd, nil, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", + "Withdrew the export of Service %s", name) + return nil +} + +// mapServiceExportToDGDRequests requeues each DGD in the export's namespace that +// exports a Service of that name, whether or not it owns the export: the DGD +// withdraws an export it created moments before losing its pool mark, and takes +// over when a user's export of the same Service is deleted. +func (r *DynamoGraphDeploymentReconciler) mapServiceExportToDGDRequests( + ctx context.Context, + obj client.Object, +) []ctrl.Request { + deployments := &nvidiacomv1beta1.DynamoGraphDeploymentList{} + if err := r.List(ctx, deployments, client.InNamespace(obj.GetNamespace())); err != nil { + log.FromContext(ctx).Error(err, "Failed to list DynamoGraphDeployments for ServiceExport event") + return nil + } + + requests := make([]ctrl.Request, 0) + for i := range deployments.Items { + if slices.Contains(multicluster.ServiceNames(deployments.Items[i].Name), obj.GetName()) { + requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(&deployments.Items[i])}) + } + } + return requests +} diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go new file mode 100644 index 000000000000..4585a712d951 --- /dev/null +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go @@ -0,0 +1,272 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package controller + +import ( + "context" + "slices" + "strings" + "testing" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + "github.com/onsi/gomega" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/uuid" + "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" +) + +// newServiceExportTestClient returns a fake client that also serves the +// ServiceExport kind, which the operator's scheme does not register. +func newServiceExportTestClient( + scheme *runtime.Scheme, + funcs interceptor.Funcs, + objects ...client.Object, +) client.WithWatch { + mapper := meta.NewDefaultRESTMapper(nil) + for gvk := range scheme.AllKnownTypes() { + mapper.Add(gvk, meta.RESTScopeNamespace) + } + mapper.Add(multicluster.ServiceExportGVK, meta.RESTScopeNamespace) + return fake.NewClientBuilder(). + WithScheme(scheme). + WithRESTMapper(mapper). + WithInterceptorFuncs(funcs). + WithObjects(objects...). + Build() +} + +func TestDGDServiceExportReconciler(t *testing.T) { + const namespace = "dynamo" + // -pool-relay is 56 characters, one more than Karmada imports; + // -frontend is 54. + longName := strings.Repeat("a", 45) + + for _, tc := range []struct { + name string + dgdName string + pool bool + implementation configv1alpha1.MCSImplementation + ownedBefore []string + terminating bool + foreign []string + wantOwned []string + wantEvents int + }{ + { + name: "a pool exports its PoolRelay and frontend Services", + dgdName: "qwen", + pool: true, + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{"qwen-pool-relay", "qwen-frontend"}, + wantEvents: 2, + }, + { + name: "exports use the Service names, with dots replaced", + dgdName: "qwen3-0.6b", + pool: true, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"qwen3-0-6b-pool-relay", "qwen3-0-6b-frontend"}, + wantEvents: 2, + }, + { + name: "removing the pool mark withdraws the exports", + dgdName: "qwen", + implementation: configv1alpha1.MCSImplementationKarmada, + ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, + wantEvents: 2, + }, + { + name: "an export already being deleted is not deleted again", + dgdName: "qwen", + implementation: configv1alpha1.MCSImplementationStandard, + ownedBefore: []string{"qwen-pool-relay"}, + terminating: true, + wantOwned: []string{"qwen-pool-relay"}, + }, + { + name: "an export a user created is neither adopted nor deleted", + dgdName: "qwen", + pool: true, + implementation: configv1alpha1.MCSImplementationStandard, + foreign: []string{"qwen-frontend"}, + wantOwned: []string{"qwen-pool-relay"}, + wantEvents: 1, + }, + { + name: "a DGD that is not a pool leaves a user's export alone", + dgdName: "qwen", + implementation: configv1alpha1.MCSImplementationStandard, + foreign: []string{"qwen-pool-relay"}, + }, + { + name: "without an MCS implementation nothing is exported", + dgdName: "qwen", + pool: true, + implementation: configv1alpha1.MCSImplementationNone, + }, + { + name: "Karmada skips a Service name it cannot import", + dgdName: longName, + pool: true, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{longName + "-frontend"}, + wantEvents: 1, + }, + { + name: "standard MCS exports the same long names", + dgdName: longName, + pool: true, + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, + wantEvents: 2, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create the DGD and the exports that exist before reconciling") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := &v1beta1.DynamoGraphDeployment{ + Name: tc.dgdName, Namespace: namespace, UID: uuid.NewUUID(), + } + if tc.pool { + dgd.Annotations = map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"} + } + objects := []client.Object{dgd} + for _, name := range tc.ownedBefore { + owned := multicluster.RenderServiceExport(namespace, name, dgd.Name) + g.Expect(controllerutil.SetControllerReference(dgd, owned, scheme)).To(gomega.Succeed()) + if tc.terminating { + owned.SetFinalizers([]string{"example.com/hold"}) + owned.SetDeletionTimestamp(&metav1.Time{Time: metav1.Now().Time}) + } + objects = append(objects, owned) + } + for _, name := range tc.foreign { + objects = append(objects, multicluster.RenderServiceExport(namespace, name, dgd.Name)) + } + fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, objects...) + + t.Log("Reconcile the exports, with the gate the configured implementation resolves to") + config := &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: tc.implementation}, + } + gates := features.Gates{ + MultiClusterExport: tc.implementation != configv1alpha1.MCSImplementationNone, + } + recorder := events.NewFakeRecorder(10) + reconciler := newDGDServiceExportReconciler(newDGDResourceSyncer(fakeClient, recorder), config, gates) + g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) + g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) + + t.Log("Reconcile again: every export is in place, so nothing is announced") + g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) + g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) + + t.Log("Check which exports the DGD controls, and that a user's export is untouched") + for _, name := range multicluster.ServiceNames(dgd.Name) { + export := multicluster.ServiceExportMetadata() + err := fakeClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, export) + switch { + case slices.Contains(tc.wantOwned, name): + g.Expect(err).NotTo(gomega.HaveOccurred(), name) + g.Expect(metav1.IsControlledBy(export, dgd)).To(gomega.BeTrue(), name) + g.Expect(export.Labels).To(gomega.HaveKeyWithValue(consts.KubeLabelDynamoGraphDeploymentName, dgd.Name)) + case slices.Contains(tc.foreign, name): + g.Expect(err).NotTo(gomega.HaveOccurred(), name) + g.Expect(metav1.IsControlledBy(export, dgd)).To(gomega.BeFalse(), name) + default: + g.Expect(apierrors.IsNotFound(err)).To(gomega.BeTrue(), "%s: %v", name, err) + } + } + }) + } +} + +func TestDGDServiceExportReconcilerToleratesAStaleCache(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("The cache has not yet seen exports that the API server already has") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := &v1beta1.DynamoGraphDeployment{ + Name: "qwen", Namespace: "dynamo", UID: uuid.NewUUID(), + Annotations: map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"}, + } + fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + return apierrors.NewAlreadyExists(multicluster.ServiceExportGVK.GroupVersion().WithResource("serviceexports").GroupResource(), obj.GetName()) + }, + }, dgd) + + t.Log("Reconcile: AlreadyExists is no error, and nothing is announced") + config := &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: configv1alpha1.MCSImplementationStandard}, + } + recorder := events.NewFakeRecorder(10) + reconciler := newDGDServiceExportReconciler( + newDGDResourceSyncer(fakeClient, recorder), + config, + features.Gates{MultiClusterExport: true}, + ) + g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) + g.Expect(recorder.Events).To(gomega.BeEmpty()) +} + +func TestMapServiceExportToDGDRequests(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create two DGDs in one namespace, one with a dot in its name, and a third elsewhere") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dotted := &v1beta1.DynamoGraphDeployment{Name: "qwen3-0.6b", Namespace: "dynamo"} + llama := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "dynamo"} + elsewhere := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "other"} + reconciler := &DynamoGraphDeploymentReconciler{ + Client: newServiceExportTestClient(scheme, interceptor.Funcs{}, dotted, llama, elsewhere), + } + + for _, tc := range []struct { + export string + want []ctrl.Request + }{ + {export: "qwen3-0-6b-pool-relay", want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(dotted)}}}, + {export: "llama-frontend", want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}}, + {export: "llama", want: []ctrl.Request{}}, + } { + t.Log("Map an event for export " + tc.export + " in namespace dynamo to the DGDs that export it") + export := multicluster.ServiceExportMetadata() + export.Namespace = "dynamo" + export.Name = tc.export + g.Expect(reconciler.mapServiceExportToDGDRequests(ctx, export)).To(gomega.Equal(tc.want), tc.export) + } +} diff --git a/deploy/operator/internal/controller/dynamographdeployment_controller.go b/deploy/operator/internal/controller/dynamographdeployment_controller.go index edfa9b17311f..1b463617f8f3 100644 --- a/deploy/operator/internal/controller/dynamographdeployment_controller.go +++ b/deploy/operator/internal/controller/dynamographdeployment_controller.go @@ -51,6 +51,7 @@ import ( "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo/epp" "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" snapshotv1alpha1 "github.com/ai-dynamo/snapshot/api/v1alpha1" ) @@ -92,6 +93,7 @@ type DynamoGraphDeploymentReconciler struct { // +kubebuilder:rbac:groups=scheduling.run.ai,resources=queues,verbs=get;list // +kubebuilder:rbac:groups=inference.networking.k8s.io,resources=inferencepools,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=networking.istio.io,resources=destinationrules,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=multicluster.x-k8s.io,resources=serviceexports,verbs=get;list;watch;create;delete // +kubebuilder:rbac:groups=nvidia.com,resources=snapshotjobs,verbs=get;list;watch;create;patch;delete // +kubebuilder:rbac:groups=nvidia.com,resources=podsnapshots,verbs=get;list;watch;patch;delete // +kubebuilder:rbac:groups=resource.k8s.io,resources=resourceclaimtemplates,verbs=get;list;watch;create;update;patch;delete @@ -383,6 +385,20 @@ func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) err GenericFunc: func(ge event.GenericEvent) bool { return false }, })) } + // A Global Router pool's exports converge once the cache sees them created or + // deleted, the DGD's own and a user's of the same Service alike. + if r.RuntimeConfig.Gate.Enabled(features.MultiClusterExport) { + ctrlBuilder = ctrlBuilder.Watches( + multicluster.ServiceExportMetadata(), + handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToDGDRequests), + builder.WithPredicates(predicate.Funcs{ + CreateFunc: func(ce event.CreateEvent) bool { return true }, + DeleteFunc: func(de event.DeleteEvent) bool { return true }, + UpdateFunc: func(ue event.UpdateEvent) bool { return false }, + GenericFunc: func(ge event.GenericEvent) bool { return false }, + }), + ) + } if r.RuntimeConfig.Gate.Enabled(features.DisaggregatedSet) { ctrlBuilder = newDisaggregatedSetWatchSetup(mgr.GetClient()).addTo(ctrlBuilder) } diff --git a/deploy/operator/internal/features/gates.go b/deploy/operator/internal/features/gates.go index feca0117b253..ac59b6f58762 100644 --- a/deploy/operator/internal/features/gates.go +++ b/deploy/operator/internal/features/gates.go @@ -148,6 +148,7 @@ var allNames = [...]Name{ VolcanoScheduler, DRA, Istio, + MultiClusterExport, GPUDiscovery, } @@ -158,16 +159,17 @@ type Gate interface { // Gates is the complete set of operator feature gates. type Gates struct { - Checkpoint bool `json:"checkpoint"` - Grove bool `json:"grove"` - LPX bool `json:"lpx"` - LWS bool `json:"lws"` - DisaggregatedSet bool `json:"disaggregatedSet"` - KaiScheduler bool `json:"kaiScheduler"` - VolcanoScheduler bool `json:"volcanoScheduler"` - DRA bool `json:"dra"` - Istio bool `json:"istio"` - GPUDiscovery bool `json:"gpuDiscovery"` + Checkpoint bool `json:"checkpoint"` + Grove bool `json:"grove"` + LPX bool `json:"lpx"` + LWS bool `json:"lws"` + DisaggregatedSet bool `json:"disaggregatedSet"` + KaiScheduler bool `json:"kaiScheduler"` + VolcanoScheduler bool `json:"volcanoScheduler"` + DRA bool `json:"dra"` + Istio bool `json:"istio"` + MultiClusterExport bool `json:"multiClusterExport"` + GPUDiscovery bool `json:"gpuDiscovery"` } // Defaults returns the default feature gates. @@ -284,6 +286,14 @@ func New( } } + // Export Global Router pools only through a configured MCS implementation whose + // ServiceExport API this cluster serves. + if config.MultiCluster.MCSImplementation != configv1alpha1.MCSImplementationNone { + if gates.MultiClusterExport, err = resolveMultiClusterExport(ctx, mgr.GetConfig()); err != nil { + return Gates{}, err + } + } + logValues := make([]any, 0, 2*len(allNames)) for _, name := range allNames { logValues = append(logValues, string(name), gates.Enabled(name)) diff --git a/deploy/operator/internal/features/gates_test.go b/deploy/operator/internal/features/gates_test.go index 7db902449817..8f9cf63f5cd2 100644 --- a/deploy/operator/internal/features/gates_test.go +++ b/deploy/operator/internal/features/gates_test.go @@ -56,16 +56,17 @@ func TestGateRegistryIsComplete(t *testing.T) { func allEnabledGates() Gates { return Gates{ - Checkpoint: true, - Grove: true, - LPX: true, - LWS: true, - DisaggregatedSet: true, - KaiScheduler: true, - VolcanoScheduler: true, - DRA: true, - Istio: true, - GPUDiscovery: true, + Checkpoint: true, + Grove: true, + LPX: true, + LWS: true, + DisaggregatedSet: true, + KaiScheduler: true, + VolcanoScheduler: true, + DRA: true, + Istio: true, + MultiClusterExport: true, + GPUDiscovery: true, } } diff --git a/deploy/operator/internal/features/multicluster.go b/deploy/operator/internal/features/multicluster.go new file mode 100644 index 000000000000..70069c48007c --- /dev/null +++ b/deploy/operator/internal/features/multicluster.go @@ -0,0 +1,38 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package features + +import ( + "context" + + "k8s.io/client-go/rest" +) + +// MultiClusterExport exports the PoolRelay and frontend Services of each DGD +// marked as a Global Router pool through the Multi-Cluster Services (MCS) API. +// +// Owner: @atchernych +// Experimental since: v1.7.0 +// Beta since: N/A +// GA since: N/A +// Configuration: multiCluster.mcsImplementation, and the +// nvidia.com/global-router-pool annotation on each DGD +// Auto-detection: N/A; API availability is verified when an implementation is configured +// Requires: an MCS implementation, such as Karmada, serving +// multicluster.x-k8s.io/v1alpha1 ServiceExport resources +// Default: false +const MultiClusterExport Name = "multiClusterExport" + +// resolveMultiClusterExport verifies API availability for a configured MCS +// implementation. A nil config retains the native discovery-unavailable error. +func resolveMultiClusterExport(ctx context.Context, config *rest.Config) (bool, error) { + // Every supported implementation reads the SIG Multicluster ServiceExport. + available, err := detectAPIAvailability(ctx, config, "multicluster.x-k8s.io", "v1alpha1", "serviceexports") + if err != nil { + return false, err + } + + return resolve(new(true), available, + "multiCluster.mcsImplementation is set but the multicluster.x-k8s.io/v1alpha1 ServiceExport API was not detected in the cluster") +} diff --git a/deploy/operator/internal/features/multicluster_test.go b/deploy/operator/internal/features/multicluster_test.go new file mode 100644 index 000000000000..f5cfc1467e26 --- /dev/null +++ b/deploy/operator/internal/features/multicluster_test.go @@ -0,0 +1,69 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package features + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/rest" +) + +func TestResolveMultiClusterExport(t *testing.T) { + const ( + mcsPath = "/apis/multicluster.x-k8s.io/v1alpha1" + missingAPI = "multiCluster.mcsImplementation is set but the multicluster.x-k8s.io/v1alpha1 ServiceExport API was not detected in the cluster" + ) + tests := []struct { + name string + status int + resource string + wantErr string + }{ + {name: "ServiceExport API present", status: http.StatusOK, resource: "serviceexports"}, + {name: "group version absent", status: http.StatusNotFound, wantErr: missingAPI}, + {name: "only ServiceImport served", status: http.StatusOK, resource: "serviceimports", wantErr: missingAPI}, + {name: "discovery forbidden", status: http.StatusForbidden, wantErr: "discover multicluster.x-k8s.io/v1alpha1 API resources: Forbidden"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Log("Serve the multicluster.x-k8s.io/v1alpha1 discovery path") + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != mcsPath { + t.Errorf("unexpected discovery path %q", r.URL.Path) + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(tt.status) + var response any = metav1.Status{Kind: "Status", APIVersion: "v1", Status: metav1.StatusFailure, Code: int32(tt.status), Message: http.StatusText(tt.status)} + if tt.status == http.StatusOK { + response = metav1.APIResourceList{ + Kind: "APIResourceList", + APIVersion: "v1", + GroupVersion: strings.TrimPrefix(r.URL.Path, "/apis/"), + APIResources: []metav1.APIResource{{Name: tt.resource}}, + } + } + if err := json.NewEncoder(w).Encode(response); err != nil { + t.Errorf("write discovery response: %v", err) + } + })) + defer server.Close() + + t.Log("Require the ServiceExport resource and preserve discovery failures") + enabled, err := resolveMultiClusterExport(t.Context(), &rest.Config{Host: server.URL}) + if enabled != (tt.wantErr == "") || (err == nil) != (tt.wantErr == "") { + t.Fatalf("resolveMultiClusterExport() = %v, %v; want enabled=%v, error=%q", enabled, err, tt.wantErr == "", tt.wantErr) + } + if err != nil && err.Error() != tt.wantErr { + t.Errorf("resolveMultiClusterExport() error = %q, want %q", err, tt.wantErr) + } + }) + } +} diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go new file mode 100644 index 000000000000..e43b1697db79 --- /dev/null +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -0,0 +1,92 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +// Package multicluster renders the ServiceExports that announce a Global Router +// pool to other clusters through the Kubernetes Multi-Cluster Services (MCS) API. +package multicluster + +import ( + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/validation" +) + +// ServiceExportGVK is the SIG Multicluster ServiceExport. Every supported +// implementation reads it in the cluster that runs the Service, Karmada included: +// its ServiceExport controller watches this kind in member clusters. Karmada +// watches a member only while its control plane has propagated at least one +// ServiceExport there, so keep one in place in each member. That setup is outside +// the operator. +var ServiceExportGVK = schema.GroupVersionKind{ + Group: "multicluster.x-k8s.io", + Version: "v1alpha1", + Kind: "ServiceExport", +} + +const ( + // PoolRelayServiceSuffix names a pool's PoolRelay Service, -pool-relay, + // as the Global Discovery Service expects by default. + PoolRelayServiceSuffix = "-pool-relay" + // FrontendServiceSuffix names a pool's frontend Service, -frontend, as + // the Global Discovery Service expects by default. The operator creates this + // Service for a frontend component named "Frontend". + FrontendServiceSuffix = "-frontend" + + // maxKarmadaServiceNameLength is the longest Service name Karmada imports. + // Karmada labels each collected slice kubernetes.io/service-name: + // derived-, and a label value has at most 63 characters. + maxKarmadaServiceNameLength = 55 +) + +// ServiceNames returns the names of a pool's PoolRelay and frontend Services, the +// Services the operator exports for a Global Router pool. Like the operator's own +// Service names, they replace the dots in a DGD name with dashes. +func ServiceNames(dgdName string) []string { + return []string{ + dynamo.NormalizeKubeResourceName(dgdName + PoolRelayServiceSuffix), + dynamo.NormalizeKubeResourceName(dgdName + FrontendServiceSuffix), + } +} + +// IsPool reports whether a DGD with these annotations is marked as a Global Router +// pool. A nil map is supported and is not a pool. +func IsPool(annotations map[string]string) bool { + return annotations[commonconsts.KubeAnnotationGlobalRouterPool] == "true" +} + +// MaxServiceNameLength returns the longest Service name that implementation can +// export with the cluster it came from. Standard implementations label imported +// EndpointSlices with their source cluster, so any Service name works. +func MaxServiceNameLength(implementation configv1alpha1.MCSImplementation) int { + if implementation == configv1alpha1.MCSImplementationKarmada { + return maxKarmadaServiceNameLength + } + return validation.DNS1035LabelMaxLength +} + +// ServiceExportMetadata returns an empty metadata-only ServiceExport, to read, +// delete or watch exports through the cache without their content. +func ServiceExportMetadata() *metav1.PartialObjectMetadata { + serviceExport := &metav1.PartialObjectMetadata{} + serviceExport.SetGroupVersionKind(ServiceExportGVK) + return serviceExport +} + +// RenderServiceExport returns the ServiceExport for the Service named name in +// namespace, labeled with the DGD that exports it. It sets no owner reference. +func RenderServiceExport(namespace, name, dgdName string) *unstructured.Unstructured { + serviceExport := &unstructured.Unstructured{} + serviceExport.SetGroupVersionKind(ServiceExportGVK) + serviceExport.SetNamespace(namespace) + serviceExport.SetName(name) + serviceExport.SetLabels(map[string]string{ + commonconsts.KubeLabelDynamoGraphDeploymentName: dgdName, + }) + return serviceExport +} diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go new file mode 100644 index 000000000000..11e27a5c14e5 --- /dev/null +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -0,0 +1,82 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +package multicluster + +import ( + "slices" + "testing" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" +) + +func TestIsPool(t *testing.T) { + for _, tc := range []struct { + name string + annotations map[string]string + want bool + }{ + {name: "no annotations"}, + {name: "marked", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "true"}, want: true}, + {name: "marked false", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "false"}}, + {name: "not a boolean", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "yes"}}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Log("Read the Global Router pool mark") + if got := IsPool(tc.annotations); got != tc.want { + t.Errorf("IsPool(%v) = %v, want %v", tc.annotations, got, tc.want) + } + }) + } +} + +func TestMaxServiceNameLength(t *testing.T) { + for _, tc := range []struct { + implementation configv1alpha1.MCSImplementation + want int + }{ + {implementation: configv1alpha1.MCSImplementationStandard, want: 63}, + {implementation: configv1alpha1.MCSImplementationKarmada, want: 55}, + } { + t.Run(string(tc.implementation), func(t *testing.T) { + t.Log("Look up the longest Service name the implementation exports with its cluster") + if got := MaxServiceNameLength(tc.implementation); got != tc.want { + t.Errorf("MaxServiceNameLength(%q) = %d, want %d", tc.implementation, got, tc.want) + } + }) + } +} + +func TestServiceNames(t *testing.T) { + for _, tc := range []struct { + dgdName string + want []string + }{ + {dgdName: "qwen", want: []string{"qwen-pool-relay", "qwen-frontend"}}, + {dgdName: "qwen3-0.6b", want: []string{"qwen3-0-6b-pool-relay", "qwen3-0-6b-frontend"}}, + } { + t.Run(tc.dgdName, func(t *testing.T) { + t.Log("Name the pool's Services as the operator names its own") + if got := ServiceNames(tc.dgdName); !slices.Equal(got, tc.want) { + t.Errorf("ServiceNames(%q) = %v, want %v", tc.dgdName, got, tc.want) + } + }) + } +} + +func TestRenderServiceExport(t *testing.T) { + t.Log("Render the export of pool qwen's PoolRelay Service") + export := RenderServiceExport("dynamo", "qwen-pool-relay", "qwen") + if got := export.GroupVersionKind(); got != ServiceExportGVK { + t.Errorf("GroupVersionKind = %v, want %v", got, ServiceExportGVK) + } + if export.GetNamespace() != "dynamo" || export.GetName() != "qwen-pool-relay" { + t.Errorf("export = %s/%s, want dynamo/qwen-pool-relay", export.GetNamespace(), export.GetName()) + } + if got := export.GetLabels()[commonconsts.KubeLabelDynamoGraphDeploymentName]; got != "qwen" { + t.Errorf("DGD label = %q, want qwen", got) + } +} diff --git a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md index 6331c345674b..9d23c833e11c 100644 --- a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md +++ b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md @@ -4169,6 +4169,24 @@ _Appears in:_ | `format` _string_ | Format is the log format (e.g., "json", "text") | json | | +#### MCSImplementation + +_Underlying type:_ _string_ + +MCSImplementation identifies an implementation of the Multi-Cluster Services API. + + + +_Appears in:_ +- [MultiClusterConfiguration](#multiclusterconfiguration) + +| Field | Description | +| --- | --- | +| `` | MCSImplementationNone exports nothing.
| +| `mcs` | MCSImplementationStandard is an implementation of the SIG Multicluster API that
imports Services into ServiceImports and EndpointSlices with the standard labels.
| +| `karmada` | MCSImplementationKarmada is Karmada, which reads the same ServiceExport in
member clusters but names imported EndpointSlices after the source cluster.
| + + #### MPIConfiguration @@ -4204,6 +4222,26 @@ _Appears in:_ | `secure` _boolean_ | Secure enables secure serving for the metrics endpoint.
nil = default to true (secure by default). | | | +#### MultiClusterConfiguration + + + +MultiClusterConfiguration configures how the operator exports Global Router +pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global +Discovery Service's Kubernetes stores. The operator exports a pool only when +MCSImplementation is set and its DGD has the nvidia.com/global-router-pool +annotation. + + + +_Appears in:_ +- [OperatorConfiguration](#operatorconfiguration) + +| Field | Description | Default | Validation | +| --- | --- | --- | --- | +| `mcsImplementation` _[MCSImplementation](#mcsimplementation)_ | MCSImplementation is the MCS implementation that imports the exports. Empty,
the default, exports nothing: Global Router pools are then discovered without
Kubernetes. Clearing it leaves earlier exports in place until their DGDs are
deleted. | | | + + #### NamespaceConfiguration @@ -4264,6 +4302,7 @@ OperatorConfiguration is the Schema for the operator configuration. | `mpi` _[MPIConfiguration](#mpiconfiguration)_ | MPI SSH secret configuration | | | | `checkpoint` _[CheckpointConfiguration](#checkpointconfiguration)_ | Checkpoint/restore configuration | | | | `discovery` _[DiscoveryConfiguration](#discoveryconfiguration)_ | Discovery backend configuration | | | +| `multiCluster` _[MultiClusterConfiguration](#multiclusterconfiguration)_ | MultiCluster configures exporting Global Router pools to other clusters. | | | | `gpu` _[GPUConfiguration](#gpuconfiguration)_ | GPU discovery configuration | | | | `logging` _[LoggingConfiguration](#loggingconfiguration)_ | Logging configuration | | | | `security` _[SecurityConfiguration](#securityconfiguration)_ | HTTP/2 and TLS settings | | | diff --git a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx index d0bd9f30f446..6653db5cb0ee 100644 --- a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx +++ b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx @@ -143,6 +143,12 @@ A shared LPX draft has a legacy minimum of `1` but no independent scaling group. Both fields retain their native hub/spoke conversion mapping. Converting between `v1alpha1` and `v1beta1` preserves the chosen form and does not migrate a deployment or select a strategy. +## Global Router pool export + +Set `metadata.annotations["nvidia.com/global-router-pool"]: "true"` to mark the DGD as a Global Router pool. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the Services `-pool-relay` and `-frontend`, with any dots in the DGD name replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. + +The DGD owns these exports: removing the annotation withdraws them, and deleting the DGD deletes them. A `ServiceExport` of the same name that the DGD does not own is left alone. With `karmada`, a Service name longer than 55 characters is not exported, because Karmada cannot import it. Without a configured implementation, the annotation has no effect. + ## Spec reference diff --git a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx index 571fd8ba74d1..46a1e33337e2 100644 --- a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx +++ b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx @@ -10,13 +10,13 @@ subtitle: Full Kubernetes CRD + operator-config API reference, generated from th > [!WARNING] > **Auto-generated from source.** This page is regenerated from the Dynamo operator CRDs by `docs/fern/scripts/gen_kubernetes_api.py` and covers **every** type across all three API packages: the deprecated `nvidia.com/v1alpha1` surface, the supported `nvidia.com/v1beta1` surface, and the operator's own `operator.config.dynamo.nvidia.com/v1alpha1` configuration. The trimmed [DGD](dynamo-graph-deployment.mdx), [DGDR](dynamo-graph-deployment-request.mdx), and [DCD](dynamo-component-deployment.mdx) references cover only user-facing `v1beta1` fields. To edit the surface, change the Go types under `deploy/operator/api/` and let CI regenerate this page. -Dynamo publishes 4 Kubernetes API packages with 208 typed CRD and config sections. +Dynamo publishes 4 Kubernetes API packages with 210 typed CRD and config sections. 71 types 76 types 32 types -29 types +31 types
@@ -5746,6 +5746,21 @@ Format is the log format (e.g., "json", "text") + +MCSImplementation identifies an implementation of the Multi-Cluster Services API. + + +**Kind:** `enum` +**Underlying type:** `string` +**Appears in:** [MultiClusterConfiguration](#multiclusterconfiguration) + +**Allowed values** + +- `` MCSImplementationNone exports nothing. +- mcs MCSImplementationStandard is an implementation of the SIG Multicluster API that imports Services into ServiceImports and EndpointSlices with the standard labels. +- karmada MCSImplementationKarmada is Karmada, which reads the same ServiceExport in member clusters but names imported EndpointSlices after the source cluster. + + MPIConfiguration holds MPI SSH secret settings. @@ -5782,6 +5797,23 @@ Secure enables secure serving for the metrics endpoint. nil = default to true (s + +MultiClusterConfiguration configures how the operator exports Global Router +pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global +Discovery Service's Kubernetes stores. The operator exports a pool only when +MCSImplementation is set and its DGD has the nvidia.com/global-router-pool +annotation. + + +**Kind:** `type` +**Appears in:** [OperatorConfiguration](#operatorconfiguration) + + +MCSImplementation is the MCS implementation that imports the exports. Empty, the default, exports nothing: Global Router pools are then discovered without Kubernetes. Clearing it leaves earlier exports in place until their DGDs are deleted. +See [MCSImplementation](#mcsimplementation). + + + NamespaceConfiguration determines operator namespace mode. @@ -5891,6 +5923,11 @@ Discovery backend configuration See [DiscoveryConfiguration](#discoveryconfiguration). + +MultiCluster configures exporting Global Router pools to other clusters. +See [MultiClusterConfiguration](#multiclusterconfiguration). + + GPU discovery configuration See [GPUConfiguration](#gpuconfiguration). From 222e7ff99eb19d2d274a8dd0112b9dd99dd8eaf4 Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Thu, 29 Oct 2026 18:36:17 -0700 Subject: [PATCH 2/9] test(docs): count the two multi-cluster operator config types `MultiClusterConfiguration` and `MCSImplementation` add two types to operator.config.dynamo.nvidia.com/v1alpha1, so the generated API reference test's baseline goes from 29 to 31. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- docs/fern/scripts/tests/test_gen_kubernetes_api.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/fern/scripts/tests/test_gen_kubernetes_api.py b/docs/fern/scripts/tests/test_gen_kubernetes_api.py index d09aa90176a2..ba5959aa86e9 100644 --- a/docs/fern/scripts/tests/test_gen_kubernetes_api.py +++ b/docs/fern/scripts/tests/test_gen_kubernetes_api.py @@ -52,7 +52,7 @@ "nvidia.com/v1alpha1": 71, "nvidia.com/v1beta1": 76, "nvidia.com/v1beta2": 35, - "operator.config.dynamo.nvidia.com/v1alpha1": 29, + "operator.config.dynamo.nvidia.com/v1alpha1": 31, } EXPECTED_OPERATOR_DEFAULT_SECTIONS = ( "Pod Specification Defaults", From 47f3c1853452e2d36980366d0926fc2e6a5be9ee Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Thu, 29 Oct 2026 19:43:23 -0700 Subject: [PATCH 3/9] fix(operator): keep pool exports from failing or stranding a DGD Fixes from a review of this PR: - A failed ServiceExport create or delete no longer marks a serving DGD Failed. The export step ran inside each workload program, whose error path sets the DGD's state to failed. It now runs in the outer reconcile: a failure is logged, recorded as a Warning event, and retried within 30 s, and the DGD's status is left to its workloads. - Removing the pool annotation now withdraws the exports even when the workloads fail or the DGD is rejected, since the step runs before those paths return early. - A Service name too long for the MCS implementation is recorded as a Warning event, not only logged. - `nvidia.com/global-router-pool` reads `true` in any case, as the DGD's other boolean annotations do. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- .../controller/dgd_component_program.go | 9 --- .../dgd_disaggregatedset_program.go | 9 --- .../internal/controller/dgd_grove_program.go | 9 --- .../dgd_service_export_reconciler.go | 3 + .../dgd_service_export_reconciler_test.go | 69 ++++++++++++++++++- .../dynamographdeployment_controller.go | 43 ++++++++++++ .../internal/multicluster/serviceexport.go | 7 +- .../multicluster/serviceexport_test.go | 1 + 8 files changed, 118 insertions(+), 32 deletions(-) diff --git a/deploy/operator/internal/controller/dgd_component_program.go b/deploy/operator/internal/controller/dgd_component_program.go index a5df983f0730..6de7c12faf45 100644 --- a/deploy/operator/internal/controller/dgd_component_program.go +++ b/deploy/operator/internal/controller/dgd_component_program.go @@ -35,7 +35,6 @@ type componentProgram struct { restartProgress *componentRestartProgressResolver workloads *componentWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler - serviceExports *dgdServiceExportReconciler lwsEnabled bool } @@ -57,7 +56,6 @@ func (r *DynamoGraphDeploymentReconciler) newComponentProgram() *componentProgra restartProgress: newComponentRestartProgressResolver(r.Client), workloads: newComponentWorkloadsReconciler(r.Client, r.Recorder, rollout), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), - serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), lwsEnabled: r.RuntimeConfig.Gate.Enabled(features.LWS), } } @@ -159,13 +157,6 @@ func (p *componentProgram) Reconcile( return programResult, fmt.Errorf("failed to reconcile scaling adapters: %w", err) } } - - // A Global Router pool's exports follow its workloads, so a failed export does - // not hold them back. - if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { - log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") - return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) - } programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_disaggregatedset_program.go b/deploy/operator/internal/controller/dgd_disaggregatedset_program.go index 8b44d378b13e..e3b58d38c168 100644 --- a/deploy/operator/internal/controller/dgd_disaggregatedset_program.go +++ b/deploy/operator/internal/controller/dgd_disaggregatedset_program.go @@ -39,7 +39,6 @@ type disaggregatedSetProgram struct { rollout *dgdWorkerRolloutReconciler workloads *disaggregatedSetWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler - serviceExports *dgdServiceExportReconciler gate features.Gate } @@ -58,7 +57,6 @@ func (r *DynamoGraphDeploymentReconciler) newDisaggregatedSetProgram() *disaggre rollout: rollout, workloads: r.newDisaggregatedSetWorkloadsReconciler(rollout), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), - serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), gate: r.RuntimeConfig.Gate, } } @@ -141,13 +139,6 @@ func (p *disaggregatedSetProgram) Reconcile( } } - // A Global Router pool's exports follow its workloads, so a failed export does - // not hold them back. - if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { - log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") - return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) - } - programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_grove_program.go b/deploy/operator/internal/controller/dgd_grove_program.go index 2d5cd59a0a26..4e560cac60ba 100644 --- a/deploy/operator/internal/controller/dgd_grove_program.go +++ b/deploy/operator/internal/controller/dgd_grove_program.go @@ -40,7 +40,6 @@ type groveProgram struct { lpxRestartProgress *lpxRestartProgressResolver workloads *groveWorkloadsReconciler scalingAdapters *dgdScalingAdaptersReconciler - serviceExports *dgdServiceExportReconciler topology *dgdGroveTopologyConditionReconciler gate features.Gate lpx *dgdLPXHandoff @@ -107,7 +106,6 @@ func (r *DynamoGraphDeploymentReconciler) newGroveProgram() *groveProgram { r.DockerSecretRetriever, ), scalingAdapters: newDGDScalingAdaptersReconciler(r.Client, r.Recorder), - serviceExports: newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate), topology: newDGDGroveTopologyConditionReconciler(r.Client), gate: r.RuntimeConfig.Gate, lpx: &dgdLPXHandoff{client: r.Client}, @@ -247,13 +245,6 @@ func (p *groveProgram) Reconcile( return programResult, fmt.Errorf("failed to reconcile scaling adapters: %w", err) } } - - // A Global Router pool's exports follow its workloads, so a failed export does - // not hold them back. - if err := p.serviceExports.Reconcile(ctx, req.DGD); err != nil { - log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") - return programResult, fmt.Errorf("failed to reconcile Global Router pool ServiceExports: %w", err) - } programResult.applyReconcileResult(req.DGD.Generation, result) return programResult, nil } diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler.go b/deploy/operator/internal/controller/dgd_service_export_reconciler.go index 664d45bec41b..673d75c59680 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler.go @@ -75,6 +75,9 @@ func (r *dgdServiceExportReconciler) Reconcile( if pool && !exportable { logger.Error(nil, "Service name is too long to export through this MCS implementation; shorten the DGD name", "service", name, "implementation", implementation, "maxLength", maxLength) + r.recorder.Eventf(dgd, nil, corev1.EventTypeWarning, "ServiceNameTooLong", "Export", + "Service %s is longer than the %d characters %s can import, so it is not exported; shorten the DGD name", + name, maxLength, implementation) } var err error diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go index 4585a712d951..a76b0aaa8f10 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go @@ -19,20 +19,25 @@ package controller import ( "context" + "errors" "slices" "strings" "testing" + "time" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/client-go/tools/events" @@ -79,6 +84,8 @@ func TestDGDServiceExportReconciler(t *testing.T) { foreign []string wantOwned []string wantEvents int + // Warnings that every reconcile repeats. + repeatedEvents int }{ { name: "a pool exports its PoolRelay and frontend Services", @@ -138,7 +145,8 @@ func TestDGDServiceExportReconciler(t *testing.T) { pool: true, implementation: configv1alpha1.MCSImplementationKarmada, wantOwned: []string{longName + "-frontend"}, - wantEvents: 1, + wantEvents: 2, + repeatedEvents: 1, }, { name: "standard MCS exports the same long names", @@ -188,9 +196,9 @@ func TestDGDServiceExportReconciler(t *testing.T) { g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) - t.Log("Reconcile again: every export is in place, so nothing is announced") + t.Log("Reconcile again: every export is in place, so only warnings repeat") g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) - g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) + g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents + tc.repeatedEvents)) t.Log("Check which exports the DGD controls, and that a user's export is untouched") for _, name := range multicluster.ServiceNames(dgd.Name) { @@ -270,3 +278,58 @@ func TestMapServiceExportToDGDRequests(t *testing.T) { g.Expect(reconciler.mapServiceExportToDGDRequests(ctx, export)).To(gomega.Equal(tc.want), tc.export) } } + +func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Mark a DGD as a pool, and refuse every ServiceExport create, as an admission policy would") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "dynamo", UID: uuid.NewUUID()} + dgd.Annotations = map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"} + refused := interceptor.Funcs{ + Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { + return apierrors.NewForbidden( + schema.GroupResource{Group: multicluster.ServiceExportGVK.Group, Resource: "serviceexports"}, + "llama-pool-relay", errors.New("denied by policy")) + }, + } + recorder := events.NewFakeRecorder(10) + reconciler := &DynamoGraphDeploymentReconciler{ + Client: newServiceExportTestClient(scheme, refused, dgd), + Recorder: recorder, + Config: &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{ + MCSImplementation: configv1alpha1.MCSImplementationStandard, + }, + }, + RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: features.Gates{MultiClusterExport: true}}, + } + + t.Log("The failure asks for a retry and is recorded as a warning, without an error that would fail the DGD") + g.Expect(reconciler.reconcilePoolExports(ctx, dgd)).To(gomega.BeTrue()) + g.Expect(recorder.Events).To(gomega.HaveLen(1)) + g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) + + t.Log("Exports that succeed ask for no retry") + reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{}, dgd) + g.Expect(reconciler.reconcilePoolExports(ctx, dgd)).To(gomega.BeFalse()) +} + +func TestRetryPoolExports(t *testing.T) { + g := gomega.NewGomegaWithT(t) + for _, tc := range []struct { + name string + result ctrl.Result + err error + want ctrl.Result + }{ + {name: "a reconcile that would not run again retries the exports", want: ctrl.Result{RequeueAfter: poolExportRetry}}, + {name: "a later retry comes sooner", result: ctrl.Result{RequeueAfter: time.Hour}, want: ctrl.Result{RequeueAfter: poolExportRetry}}, + {name: "a sooner retry stays", result: ctrl.Result{RequeueAfter: time.Second}, want: ctrl.Result{RequeueAfter: time.Second}}, + {name: "an error is retried anyway", err: errors.New("failed"), want: ctrl.Result{}}, + } { + t.Log(tc.name) + g.Expect(retryPoolExports(tc.result, tc.err)).To(gomega.Equal(tc.want), tc.name) + } +} diff --git a/deploy/operator/internal/controller/dynamographdeployment_controller.go b/deploy/operator/internal/controller/dynamographdeployment_controller.go index 1b463617f8f3..2422752a7792 100644 --- a/deploy/operator/internal/controller/dynamographdeployment_controller.go +++ b/deploy/operator/internal/controller/dynamographdeployment_controller.go @@ -22,6 +22,7 @@ import ( "errors" "fmt" "strings" + "time" "github.com/ai-dynamo/dynamo/deploy/operator/internal/secret" @@ -134,6 +135,13 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, err } + // A Global Router pool's exports follow its annotation whatever happens to its + // workloads below. A failed export never fails the DGD: it is logged, recorded + // as an event, and retried. + if r.reconcilePoolExports(ctx, dynamoDeployment) { + defer func() { result = retryPoolExports(result, err) }() + } + // Reject stored LPX graphs before provider selection or any workload writes. if dynamoDeployment.HasLPXComponent() && !r.RuntimeConfig.Gate.Enabled(features.LPX) { programResult := newWorkloadProgramResult(dynamoDeployment) @@ -267,6 +275,41 @@ func workloadRoutingAnnotationsChanged(update event.UpdateEvent) bool { annotationValue(oldDGD, consts.KubeAnnotationEnableGrove) != annotationValue(newDGD, consts.KubeAnnotationEnableGrove) } +// poolExportRetry is how soon a DGD whose pool exports failed is reconciled again. +const poolExportRetry = 30 * time.Second + +// reconcilePoolExports converges a Global Router pool's ServiceExports and reports +// whether that failed, so that the DGD is reconciled again. A failure is logged +// and recorded as a Warning event. +func (r *DynamoGraphDeploymentReconciler) reconcilePoolExports( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, +) bool { + exports := newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate) + err := exports.Reconcile(ctx, dgd) + if err == nil { + return false + } + log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") + if r.Recorder != nil { + r.Recorder.Eventf(dgd, nil, corev1.EventTypeWarning, "PoolExportFailed", "Export", + "Failed to reconcile Global Router pool ServiceExports: %v", err) + } + return true +} + +// retryPoolExports makes a reconcile run again within poolExportRetry, unless it +// returned an error, which is retried anyway, or already runs again sooner. +func retryPoolExports(result ctrl.Result, err error) ctrl.Result { + if err != nil { + return result + } + if result.RequeueAfter == 0 || result.RequeueAfter > poolExportRetry { + result.RequeueAfter = poolExportRetry + } + return result +} + // SetupWithManager sets up the controller with the Manager. func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) error { if err := mgr.GetFieldIndexer().IndexField( diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index e43b1697db79..2dcaf032da9b 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -8,6 +8,8 @@ package multicluster import ( + "strings" + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" @@ -55,9 +57,10 @@ func ServiceNames(dgdName string) []string { } // IsPool reports whether a DGD with these annotations is marked as a Global Router -// pool. A nil map is supported and is not a pool. +// pool: `true` in any case, as the DGD's other boolean annotations are read. A nil +// map is supported and is not a pool. func IsPool(annotations map[string]string) bool { - return annotations[commonconsts.KubeAnnotationGlobalRouterPool] == "true" + return strings.EqualFold(annotations[commonconsts.KubeAnnotationGlobalRouterPool], "true") } // MaxServiceNameLength returns the longest Service name that implementation can diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go index 11e27a5c14e5..7bd3ae78d48e 100644 --- a/deploy/operator/internal/multicluster/serviceexport_test.go +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -21,6 +21,7 @@ func TestIsPool(t *testing.T) { }{ {name: "no annotations"}, {name: "marked", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "true"}, want: true}, + {name: "marked in another case", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "True"}, want: true}, {name: "marked false", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "false"}}, {name: "not a boolean", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "yes"}}, } { From 7a1a90117d56196f4018a70f8046227e30f7021b Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Thu, 29 Oct 2026 21:03:57 -0700 Subject: [PATCH 4/9] refactor(operator): name the PoolRelay Service in one place The DynamoPoolRelay controller from main names its Service -pool-relay, the name the pool exports use. It now builds that name from multicluster.PoolRelayServiceSuffix, so the export cannot point at a Service the relay does not create. The DGD page and the suffix's comment say which resource creates each exported Service. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- deploy/operator/internal/controller/poolrelay/controller.go | 3 ++- deploy/operator/internal/multicluster/serviceexport.go | 3 ++- .../pages/reference/kubernetes-api/dynamo-graph-deployment.mdx | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/deploy/operator/internal/controller/poolrelay/controller.go b/deploy/operator/internal/controller/poolrelay/controller.go index 31b59b4c4135..0afd94c49c29 100644 --- a/deploy/operator/internal/controller/poolrelay/controller.go +++ b/deploy/operator/internal/controller/poolrelay/controller.go @@ -23,6 +23,7 @@ import ( commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" @@ -450,7 +451,7 @@ func sourceJSON(namespaces []string, connectionRevision string) string { return string(encoded) } -func serviceName(dgdName string) string { return dgdName + "-pool-relay" } +func serviceName(dgdName string) string { return dgdName + multicluster.PoolRelayServiceSuffix } func render(relay *v1alpha1.DynamoPoolRelay, namespaces []string) ([]client.Object, *appsv1.Deployment, error) { name := serviceName(relay.Name) diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 2dcaf032da9b..17e05337ac6d 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -33,7 +33,8 @@ var ServiceExportGVK = schema.GroupVersionKind{ const ( // PoolRelayServiceSuffix names a pool's PoolRelay Service, -pool-relay, - // as the Global Discovery Service expects by default. + // as the Global Discovery Service expects by default. The operator creates this + // Service for a DynamoPoolRelay named after the DGD. PoolRelayServiceSuffix = "-pool-relay" // FrontendServiceSuffix names a pool's frontend Service, -frontend, as // the Global Discovery Service expects by default. The operator creates this diff --git a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx index 0531f1c437d3..90f7cba6e53a 100644 --- a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx +++ b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx @@ -145,7 +145,7 @@ Both fields retain their native hub/spoke conversion mapping. Converting between ## Global Router pool export -Set `metadata.annotations["nvidia.com/global-router-pool"]: "true"` to mark the DGD as a Global Router pool. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the Services `-pool-relay` and `-frontend`, with any dots in the DGD name replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. +Set `metadata.annotations["nvidia.com/global-router-pool"]: "true"` to mark the DGD as a Global Router pool. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the Services `-pool-relay` and `-frontend`, with any dots in the DGD name replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. A [`DynamoPoolRelay`](../../kubernetes/kv-aware-routing/pool-relay.md) with the DGD's name creates `-pool-relay`, and the operator creates `-frontend` for a frontend component named `Frontend`. The DGD owns these exports: removing the annotation withdraws them, and deleting the DGD deletes them. A `ServiceExport` of the same name that the DGD does not own is left alone. With `karmada`, a Service name longer than 55 characters is not exported, because Karmada cannot import it. Without a configured implementation, the annotation has no effect. From 03be5a0a3df14480d61d340f9a08139ea6c48c64 Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Fri, 30 Oct 2026 11:18:11 -0700 Subject: [PATCH 5/9] feat(operator): mark pools by their DynamoPoolRelay and report their exports A DGD is now a Global Router pool while a DynamoPoolRelay of the same name exists, in place of the nvidia.com/global-router-pool annotation, so the PoolRelay Service that the export names always has a creator. The operator exports the Service of each frontend component under the name it gives that Service, not a fixed -frontend, and withdraws the export of a frontend that was renamed or removed. A new GlobalRouterPoolExported condition on the DGD reports the exports: Exported, ExportConflict, ExportPending, ExportRejected, InvalidServiceName, ServiceNameTooLong, NoFrontend, ClusterIDMismatch and ExportFailed. It reads the Valid, Ready and Conflict conditions that the MCS implementation writes (KEP-1645): Ready False with the reason Pending is still pending, a conflict the implementation resolved keeps the pool exported, and a standard implementation counts as pending until it reports. Every status write carries the condition, including early failures, so it never fails the DGD; a newly raised problem is also a Warning event, and a failure before the DGD is known to be a pool leaves the condition as it was. With standard MCS, the operator compares the DynamoPoolRelay's clusterId with the cluster's ID in its ClusterSet, read from the About API's ClusterProperty without the cache at most once a minute, and skipped without the property or the permission to read it. ServiceExports are read through the manager's cache: its client reads unstructured objects from the API server. The DGD controller now watches DynamoPoolRelays and passes ServiceExport updates that change their status or start their deletion. The PoolRelay controller names its Service with multicluster.PoolRelayServiceName, the name the export uses. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- deploy/helm/charts/platform/README.md | 2 +- .../components/operator/files/role.yaml | 6 + deploy/helm/charts/platform/values.yaml | 6 +- deploy/operator/api/config/v1alpha1/types.go | 3 +- deploy/operator/internal/consts/consts.go | 5 - .../dgd_service_export_reconciler.go | 381 ++++++-- .../dgd_service_export_reconciler_test.go | 864 +++++++++++++++--- .../controller/dgd_workload_program_test.go | 2 +- .../dynamographdeployment_controller.go | 126 ++- .../controller/poolrelay/controller.go | 2 +- deploy/operator/internal/controller/setup.go | 3 + .../internal/features/multicluster.go | 6 +- .../internal/multicluster/serviceexport.go | 254 ++++- .../multicluster/serviceexport_test.go | 332 ++++++- .../kubernetes/kv-aware-routing/pool-relay.md | 11 + .../additional-resources/api-reference-k8s.md | 3 +- .../dynamo-graph-deployment.mdx | 20 +- .../kubernetes-api/full-api-reference.mdx | 3 +- 18 files changed, 1751 insertions(+), 278 deletions(-) diff --git a/deploy/helm/charts/platform/README.md b/deploy/helm/charts/platform/README.md index 82f81ef616aa..5d181d7ef970 100644 --- a/deploy/helm/charts/platform/README.md +++ b/deploy/helm/charts/platform/README.md @@ -594,7 +594,7 @@ Kubernetes: `>=1.30.0-0` | dynamo-operator.dynamo.serviceMesh.istio.clientCertificate | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS certificate used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.privateKey | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS private key used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.caCertificates | string | `""` | Optional path (in the istio-proxy sidecar's filesystem) to CA certificates used to verify the EPP server certificate. Used only when tlsMode is "MUTUAL". | -| dynamo-operator.dynamo.multiCluster.mcsImplementation | string | `""` | MCS implementation that imports the exports: "mcs" for the SIG Multicluster API or "karmada". For each DGD annotated `nvidia.com/global-router-pool: "true"`, the operator then creates a ServiceExport for `-pool-relay` and `-frontend`. Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this cluster. Empty exports nothing; clearing it leaves earlier exports in place until their DGDs are deleted. | +| dynamo-operator.dynamo.multiCluster.mcsImplementation | string | `""` | MCS implementation that imports the exports: "mcs" for the SIG Multicluster API or "karmada". For each DGD with a DynamoPoolRelay of the same name, the operator then creates a ServiceExport for `-pool-relay` and for each frontend component's Service. Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this cluster. Empty exports nothing; clearing it leaves earlier exports in place until their DGDs are deleted. | | dynamo-operator.dynamo.metrics.prometheusEndpoint | string | `""` | Endpoint that services can use to retrieve metrics. If set, dynamo operator will automatically inject the PROMETHEUS_ENDPOINT environment variable into services it manages. Users can override the value of the PROMETHEUS_ENDPOINT environment variable by modifying the corresponding deployment's environment variables | | dynamo-operator.dynamo.mpiRun.secretName | string | `"mpi-run-ssh-secret"` | Name of the secret containing the SSH key for MPI Run | | dynamo-operator.webhook.certificateSecret.name | string | `"webhook-server-cert"` | Name of the Kubernetes secret containing webhook TLS certificates. The secret must contain three keys: tls.crt (server certificate), tls.key (server private key), and ca.crt (Certificate Authority certificate). | diff --git a/deploy/helm/charts/platform/components/operator/files/role.yaml b/deploy/helm/charts/platform/components/operator/files/role.yaml index 5cf891020191..b522ff9693ec 100644 --- a/deploy/helm/charts/platform/components/operator/files/role.yaml +++ b/deploy/helm/charts/platform/components/operator/files/role.yaml @@ -76,6 +76,12 @@ rules: - patch - update - watch +- apiGroups: + - about.k8s.io + resources: + - clusterproperties + verbs: + - get - apiGroups: - apiextensions.k8s.io resources: diff --git a/deploy/helm/charts/platform/values.yaml b/deploy/helm/charts/platform/values.yaml index 4599934d8114..6df998f4c889 100644 --- a/deploy/helm/charts/platform/values.yaml +++ b/deploy/helm/charts/platform/values.yaml @@ -251,9 +251,9 @@ dynamo-operator: # Services (MCS) API, for the Global Discovery Service's Kubernetes stores. multiCluster: # -- MCS implementation that imports the exports: "mcs" for the SIG - # Multicluster API or "karmada". For each DGD annotated - # `nvidia.com/global-router-pool: "true"`, the operator then creates a - # ServiceExport for `-pool-relay` and `-frontend`. + # Multicluster API or "karmada". For each DGD with a DynamoPoolRelay of + # the same name, the operator then creates a ServiceExport for + # `-pool-relay` and for each frontend component's Service. # Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this # cluster. Empty exports nothing; clearing it leaves earlier exports in # place until their DGDs are deleted. diff --git a/deploy/operator/api/config/v1alpha1/types.go b/deploy/operator/api/config/v1alpha1/types.go index 76a92bf00153..dcb74f94e45d 100644 --- a/deploy/operator/api/config/v1alpha1/types.go +++ b/deploy/operator/api/config/v1alpha1/types.go @@ -403,8 +403,7 @@ type DiscoveryConfiguration struct { // MultiClusterConfiguration configures how the operator exports Global Router // pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global // Discovery Service's Kubernetes stores. The operator exports a pool only when -// MCSImplementation is set and its DGD has the nvidia.com/global-router-pool -// annotation. +// MCSImplementation is set and a DynamoPoolRelay with its DGD's name exists. type MultiClusterConfiguration struct { // MCSImplementation is the MCS implementation that imports the exports. Empty, // the default, exports nothing: Global Router pools are then discovered without diff --git a/deploy/operator/internal/consts/consts.go b/deploy/operator/internal/consts/consts.go index 732fc7b1af20..f926af4a9db7 100644 --- a/deploy/operator/internal/consts/consts.go +++ b/deploy/operator/internal/consts/consts.go @@ -61,11 +61,6 @@ const ( // when the namespace carries istio-injection=enabled. KubeAnnotationIstioSidecarInject = "sidecar.istio.io/inject" - // KubeAnnotationGlobalRouterPool set to "true" marks a DGD as a Global Router - // pool. When the operator is configured with a Multi-Cluster Services - // implementation, it exports the pool's PoolRelay and frontend Services. - KubeAnnotationGlobalRouterPool = "nvidia.com/global-router-pool" - KubeAnnotationDisableImagePullSecretDiscovery = "nvidia.com/disable-image-pull-secret-discovery" KubeAnnotationDynamoDiscoveryBackend = "nvidia.com/dynamo-discovery-backend" KubeAnnotationDynamoKubeDiscoveryMode = "nvidia.com/dynamo-kube-discovery-mode" diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler.go b/deploy/operator/internal/controller/dgd_service_export_reconciler.go index 673d75c59680..20dd2f337df7 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler.go @@ -20,95 +20,198 @@ package controller import ( "context" "fmt" + "maps" "slices" + "strings" + "sync" + "time" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + nvidiacomv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" nvidiacomv1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/equality" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" "sigs.k8s.io/controller-runtime/pkg/log" ) +// globalRouterPoolExportedConditionType reports on a Global Router pool's DGD +// whether its Services are exported, as far as the operator and the MCS +// implementation can tell. +const globalRouterPoolExportedConditionType = "GlobalRouterPoolExported" + +// Reasons of the GlobalRouterPoolExported condition. +const ( + reasonPoolExported = "Exported" + reasonPoolExportConflict = "ExportConflict" + reasonPoolExportPending = "ExportPending" + reasonPoolExportFailed = "ExportFailed" + reasonPoolInvalidServiceName = "InvalidServiceName" + reasonPoolServiceNameTooLong = "ServiceNameTooLong" + reasonPoolNoFrontend = "NoFrontend" + reasonPoolClusterIDMismatch = "ClusterIDMismatch" + reasonPoolExportRejected = "ExportRejected" +) + +// poolProblemReasons are the reasons of the GlobalRouterPoolExported condition +// that are recorded as a Warning event when they are newly raised. A failed +// reconcile records its own event. +var poolProblemReasons = []string{ + reasonPoolExportConflict, + reasonPoolInvalidServiceName, + reasonPoolServiceNameTooLong, + reasonPoolNoFrontend, + reasonPoolClusterIDMismatch, + reasonPoolExportRejected, +} + +// clusterSetIDTTL is how long the operator reuses the cluster's ID in its +// ClusterSet before reading it again. +const clusterSetIDTTL = time.Minute + // dgdServiceExportReconciler exports the PoolRelay and frontend Services of a DGD -// marked as a Global Router pool through the configured Multi-Cluster Services -// implementation, and withdraws the exports when the mark is removed. It creates -// and deletes only ServiceExports that the DGD controls. +// that is a Global Router pool through the configured Multi-Cluster Services +// implementation, and withdraws the exports when it stops being one. A DGD is a +// pool while a DynamoPoolRelay of the same name exists. It creates and deletes +// only ServiceExports that the DGD controls. type dgdServiceExportReconciler struct { dgdResourceSyncer config *configv1alpha1.OperatorConfiguration gate features.Gate + // exports reads ServiceExports, which are unstructured: the manager's client + // would read them from the API server, so this is the manager's cache, filled + // by the controller's ServiceExport watch. + exports client.Reader + // clusterSetID returns the cluster's ID in its ClusterSet, or "" when it is + // unknown, which skips the check of the pool's cluster ID. + clusterSetID func(context.Context) string } func newDGDServiceExportReconciler( syncer dgdResourceSyncer, config *configv1alpha1.OperatorConfiguration, gate features.Gate, + exports client.Reader, + clusterSetID func(context.Context) string, ) *dgdServiceExportReconciler { - return &dgdServiceExportReconciler{dgdResourceSyncer: syncer, config: config, gate: gate} + return &dgdServiceExportReconciler{ + dgdResourceSyncer: syncer, config: config, gate: gate, exports: exports, clusterSetID: clusterSetID, + } } -// Reconcile converges the DGD's ServiceExports on its pool mark. Without a -// configured MCS implementation the operator exports nothing and leaves existing -// exports alone. +// Reconcile converges the DGD's ServiceExports on whether it is a pool, and +// returns the GlobalRouterPoolExported condition for the DGD, or nil when the DGD +// is not a pool or nothing is exported. A failure returns the error with an +// ExportFailed condition when the DGD is a pool, or with nil when that is +// unknown. Without a configured MCS implementation the operator exports nothing +// and leaves existing exports alone. func (r *dgdServiceExportReconciler) Reconcile( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, -) error { +) (*metav1.Condition, error) { if !r.gate.Enabled(features.MultiClusterExport) { - return nil + return nil, nil + } + relay, err := r.poolRelay(ctx, dgd) + if err != nil { + return nil, err } + condition, err := r.reconcileExports(ctx, dgd, relay) + if err != nil && relay != nil { + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolExportFailed, "%v", err), err + } + return condition, err +} - // Export each of the pool's Services, or withdraw the export. - logger := log.FromContext(ctx) +// unexportedService is a pool Service that the implementation cannot export. +type unexportedService struct { + name string + problem multicluster.NameProblem + detail string +} + +// reconcileExports exports the Services of the pool that relay makes the DGD, +// withdraws the DGD's other exports, and returns the pool's condition. A nil +// relay withdraws every export the DGD controls. +func (r *dgdServiceExportReconciler) reconcileExports( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + relay *nvidiacomv1alpha1.DynamoPoolRelay, +) (*metav1.Condition, error) { implementation := r.config.MultiCluster.MCSImplementation - pool := multicluster.IsPool(dgd.Annotations) - maxLength := multicluster.MaxServiceNameLength(implementation) - for _, name := range multicluster.ServiceNames(dgd.Name) { - // A Service the implementation cannot import with its cluster is not exported. - exportable := len(name) <= maxLength - if pool && !exportable { - logger.Error(nil, "Service name is too long to export through this MCS implementation; shorten the DGD name", - "service", name, "implementation", implementation, "maxLength", maxLength) - r.recorder.Eventf(dgd, nil, corev1.EventTypeWarning, "ServiceNameTooLong", "Export", - "Service %s is longer than the %d characters %s can import, so it is not exported; shorten the DGD name", - name, maxLength, implementation) - } + existing, err := multicluster.ListExports(ctx, r.exports, dgd.Namespace, implementation) + if err != nil { + return nil, err + } - var err error - if pool && exportable { - err = r.export(ctx, dgd, name) - } else { - err = r.withdraw(ctx, dgd, name) + // A pool exports each of its Services that the implementation can export. An + // existing export, the DGD's or a user's, needs nothing. + var exported []string + var unexported []unexportedService + if relay != nil { + for _, name := range multicluster.ServiceNames(dgd) { + if problem, detail := multicluster.CheckServiceName(implementation, name); problem != multicluster.NameExportable { + unexported = append(unexported, unexportedService{name: name, problem: problem, detail: detail}) + continue + } + if _, found := existing[name]; !found { + if err := r.export(ctx, dgd, name); err != nil { + return nil, err + } + } + exported = append(exported, name) } - if err != nil { - return err + } + + // Withdraw the DGD's exports of Services it no longer exports: all of them once + // it is no longer a pool, and a renamed or removed frontend's. + for _, name := range slices.Sorted(maps.Keys(existing)) { + if !slices.Contains(exported, name) { + if err := r.withdraw(ctx, dgd, existing[name]); err != nil { + return nil, err + } } } - return nil + + if relay == nil { + return nil, nil + } + return r.condition(ctx, dgd, relay, exported, unexported, existing), nil } -// export creates the ServiceExport for the Service named name unless one exists. -// An existing ServiceExport is left as it is, including one a user created. -func (r *dgdServiceExportReconciler) export( +// poolRelay returns the DynamoPoolRelay that makes the DGD a Global Router pool, +// or nil when there is none or it is being deleted. +func (r *dgdServiceExportReconciler) poolRelay( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, - name string, -) error { - // An existing export, the DGD's or a user's, needs nothing. - err := r.Get(ctx, types.NamespacedName{Namespace: dgd.Namespace, Name: name}, multicluster.ServiceExportMetadata()) - if err == nil { - return nil +) (*nvidiacomv1alpha1.DynamoPoolRelay, error) { + relay := &nvidiacomv1alpha1.DynamoPoolRelay{} + err := r.Get(ctx, client.ObjectKeyFromObject(dgd), relay) + if apierrors.IsNotFound(err) { + return nil, nil + } + if err != nil { + return nil, fmt.Errorf("failed to get DynamoPoolRelay %s: %w", dgd.Name, err) } - if !apierrors.IsNotFound(err) { - return fmt.Errorf("failed to get ServiceExport %s: %w", name, err) + if !relay.DeletionTimestamp.IsZero() { + return nil, nil } + return relay, nil +} +// export creates the DGD's ServiceExport for the Service named name. +func (r *dgdServiceExportReconciler) export( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + name string, +) error { // Owned by the DGD, the export is deleted with it. serviceExport := multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name) if err := ctrl.SetControllerReference(dgd, serviceExport, r.Scheme()); err != nil { @@ -127,42 +230,163 @@ func (r *dgdServiceExportReconciler) export( return nil } -// withdraw deletes the ServiceExport for the Service named name if the DGD -// controls it and it is not already being deleted. +// withdraw deletes export if the DGD controls it and it is not already being +// deleted. func (r *dgdServiceExportReconciler) withdraw( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, - name string, + export multicluster.Export, ) error { - // Only an export the DGD controls is withdrawn. - existing := multicluster.ServiceExportMetadata() - err := r.Get(ctx, types.NamespacedName{Namespace: dgd.Namespace, Name: name}, existing) - if apierrors.IsNotFound(err) { - return nil - } - if err != nil { - return fmt.Errorf("failed to get ServiceExport %s: %w", name, err) - } - if !metav1.IsControlledBy(existing, dgd) || existing.DeletionTimestamp != nil { + if export.ControllerUID != dgd.UID || export.Deleting { return nil } // The UID precondition deletes exactly the export just read. - if err := r.Delete(ctx, existing, client.Preconditions{UID: &existing.UID}); err != nil { + serviceExport := multicluster.NewServiceExport() + serviceExport.SetNamespace(dgd.Namespace) + serviceExport.SetName(export.Name) + if err := r.Delete(ctx, serviceExport, client.Preconditions{UID: &export.UID}); err != nil { if apierrors.IsNotFound(err) { return nil } - return fmt.Errorf("failed to delete ServiceExport %s: %w", name, err) + return fmt.Errorf("failed to delete ServiceExport %s: %w", export.Name, err) } r.recorder.Eventf(dgd, nil, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", - "Withdrew the export of Service %s", name) + "Withdrew the export of Service %s", export.Name) return nil } +// condition returns the pool's GlobalRouterPoolExported condition: the first +// problem that the operator or the MCS implementation finds, else whether every +// export is imported. existing holds the exports read before this reconcile +// created any. +func (r *dgdServiceExportReconciler) condition( + ctx context.Context, + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + relay *nvidiacomv1alpha1.DynamoPoolRelay, + exported []string, + unexported []unexportedService, + existing map[string]multicluster.Export, +) *metav1.Condition { + implementation := r.config.MultiCluster.MCSImplementation + if names, detail := unexportedNames(unexported, multicluster.NameInvalid); len(names) > 0 { + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolInvalidServiceName, + "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail) + } + if names, detail := unexportedNames(unexported, multicluster.NameTooLong); len(names) > 0 { + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolServiceNameTooLong, + "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", ")) + } + if len(multicluster.FrontendServiceNames(dgd)) == 0 { + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolNoFrontend, + "The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests") + } + + // A standard implementation labels the pool's imports with the cluster's ID + // in its ClusterSet, and the Global Discovery Service joins the pool's + // registration to them only under the same cluster ID. + if multicluster.UsesClusterSetID(implementation) && r.clusterSetID != nil { + if clusterID := r.clusterSetID(ctx); clusterID != "" && clusterID != relay.Spec.ClusterID { + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolClusterIDMismatch, + "DynamoPoolRelay %s has clusterId %q, but ClusterProperty %s gives this cluster the ID %q in its ClusterSet; "+ + "the Global Discovery Service lists the pool only when they match", + relay.Name, relay.Spec.ClusterID, multicluster.ClusterIDProperty, clusterID) + } + } + + var pending, conflicting []string + for _, name := range exported { + export, found := existing[name] + switch { + case !found || export.Deleting: + // Created by this reconcile, or waiting to be replaced. Its next watch + // event requeues the DGD. + pending = append(pending, name) + case export.State == multicluster.ExportRejected: + return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolExportRejected, + "%s reports on the export of Service %s: %s", implementation, name, export.Detail) + case export.State == multicluster.ExportPending: + pending = append(pending, name) + case export.State == multicluster.ExportConflicting: + conflicting = append(conflicting, fmt.Sprintf("%s (%s)", name, export.Detail)) + } + } + if len(pending) > 0 { + return newPoolCondition(dgd, metav1.ConditionUnknown, reasonPoolExportPending, + "Waiting for %s to import %s", implementation, strings.Join(pending, ", ")) + } + if len(conflicting) > 0 { + return newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExportConflict, + "Exported %s through %s, which resolves a conflict with another cluster's export of %s", + strings.Join(exported, ", "), implementation, strings.Join(conflicting, ", ")) + } + return newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExported, + "Exported %s through %s", strings.Join(exported, ", "), implementation) +} + +// unexportedNames returns the names of the services with problem, and the first +// one's detail. +func unexportedNames(services []unexportedService, problem multicluster.NameProblem) ([]string, string) { + var names []string + detail := "" + for _, service := range services { + if service.problem != problem { + continue + } + if len(names) == 0 { + detail = service.detail + } + names = append(names, service.name) + } + return names, detail +} + +// newPoolCondition returns the DGD's GlobalRouterPoolExported condition. +func newPoolCondition( + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + status metav1.ConditionStatus, + reason, format string, + args ...any, +) *metav1.Condition { + return &metav1.Condition{ + Type: globalRouterPoolExportedConditionType, + Status: status, + ObservedGeneration: dgd.Generation, + Reason: reason, + Message: fmt.Sprintf(format, args...), + } +} + +// clusterSetIDs keeps the cluster's ID in its ClusterSet, read through the API +// server at most once per clusterSetIDTTL: the cache does not hold it, and it +// rarely changes. +type clusterSetIDs struct { + mu sync.Mutex + id string + readAt time.Time +} + +// get returns the cluster's ID in its ClusterSet, or "" when it is unknown. A +// failed read is not kept, so the next call reads again. +func (c *clusterSetIDs) get(ctx context.Context, reader client.Reader, now time.Time) string { + c.mu.Lock() + defer c.mu.Unlock() + if !c.readAt.IsZero() && now.Sub(c.readAt) < clusterSetIDTTL { + return c.id + } + id, err := multicluster.ReadClusterID(ctx, reader) + if err != nil { + log.FromContext(ctx).V(1).Info("Failed to read the cluster's ID in its ClusterSet; skipping the check", "error", err) + return "" + } + c.id, c.readAt = id, now + return id +} + // mapServiceExportToDGDRequests requeues each DGD in the export's namespace that -// exports a Service of that name, whether or not it owns the export: the DGD -// withdraws an export it created moments before losing its pool mark, and takes -// over when a user's export of the same Service is deleted. +// controls the export or exports a Service of its name: the DGD withdraws an +// export it no longer needs, takes over when a user's export of the same Service +// is deleted, and follows what the MCS implementation reports. func (r *DynamoGraphDeploymentReconciler) mapServiceExportToDGDRequests( ctx context.Context, obj client.Object, @@ -173,11 +397,40 @@ func (r *DynamoGraphDeploymentReconciler) mapServiceExportToDGDRequests( return nil } + owner := metav1.GetControllerOf(obj) requests := make([]ctrl.Request, 0) for i := range deployments.Items { - if slices.Contains(multicluster.ServiceNames(deployments.Items[i].Name), obj.GetName()) { - requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(&deployments.Items[i])}) + dgd := &deployments.Items[i] + controls := owner != nil && owner.UID == dgd.UID + if controls || slices.Contains(multicluster.ServiceNames(dgd), obj.GetName()) { + requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) } } return requests } + +// mapPoolRelayToDGDRequests requeues the DGD that a DynamoPoolRelay makes a +// Global Router pool: the one with the same name and namespace. +func mapPoolRelayToDGDRequests(_ context.Context, obj client.Object) []ctrl.Request { + return []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(obj)}} +} + +// serviceExportStateChanged passes the ServiceExport updates that can change a +// pool's condition: a new status from the MCS implementation, or the start of a +// deletion. +func serviceExportStateChanged(update event.UpdateEvent) bool { + oldExport, oldOK := update.ObjectOld.(*unstructured.Unstructured) + newExport, newOK := update.ObjectNew.(*unstructured.Unstructured) + if !oldOK || !newOK { + return false + } + return !equality.Semantic.DeepEqual(oldExport.Object["status"], newExport.Object["status"]) || + oldExport.GetDeletionTimestamp().IsZero() != newExport.GetDeletionTimestamp().IsZero() +} + +// poolRelayChanged passes the DynamoPoolRelay updates that can change its pool: +// a new spec, such as its cluster ID, or the start of a deletion. +func poolRelayChanged(update event.UpdateEvent) bool { + return update.ObjectOld.GetGeneration() != update.ObjectNew.GetGeneration() || + update.ObjectOld.GetDeletionTimestamp().IsZero() != update.ObjectNew.GetDeletionTimestamp().IsZero() +} diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go index a76b0aaa8f10..a4fb31ccedfc 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go @@ -20,12 +20,12 @@ package controller import ( "context" "errors" - "slices" "strings" "testing" "time" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" @@ -36,9 +36,9 @@ import ( apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/client-go/tools/events" ctrl "sigs.k8s.io/controller-runtime" @@ -46,10 +46,13 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" "sigs.k8s.io/controller-runtime/pkg/client/interceptor" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/reconcile" ) // newServiceExportTestClient returns a fake client that also serves the -// ServiceExport kind, which the operator's scheme does not register. +// ServiceExport and ClusterProperty kinds, which the operator's scheme does not +// register, and the DGD's status subresource. func newServiceExportTestClient( scheme *runtime.Scheme, funcs interceptor.Funcs, @@ -60,194 +63,544 @@ func newServiceExportTestClient( mapper.Add(gvk, meta.RESTScopeNamespace) } mapper.Add(multicluster.ServiceExportGVK, meta.RESTScopeNamespace) + mapper.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) return fake.NewClientBuilder(). WithScheme(scheme). WithRESTMapper(mapper). + WithStatusSubresource(&v1beta1.DynamoGraphDeployment{}). WithInterceptorFuncs(funcs). WithObjects(objects...). Build() } +// listServiceExports returns the ServiceExports in namespace. +func listServiceExports(t *testing.T, c client.Client, namespace string) []unstructured.Unstructured { + t.Helper() + list := &unstructured.UnstructuredList{} + list.SetGroupVersionKind(multicluster.ServiceExportGVK.GroupVersion().WithKind("ServiceExportList")) + if err := c.List(context.Background(), list, client.InNamespace(namespace)); err != nil { + t.Fatalf("list ServiceExports: %v", err) + } + return list.Items +} + +// newPoolTestDGD returns a DGD named name in namespace dynamo with a decode +// worker and a frontend component for each of frontends. +func newPoolTestDGD(name string, frontends ...string) *v1beta1.DynamoGraphDeployment { + dgd := &v1beta1.DynamoGraphDeployment{Name: name, Namespace: "dynamo", UID: uuid.NewUUID()} + dgd.Spec.Components = []v1beta1.DynamoComponentDeploymentSharedSpec{ + {ComponentName: "VllmDecodeWorker", ComponentType: v1beta1.ComponentTypeDecode}, + } + for _, frontend := range frontends { + dgd.Spec.Components = append(dgd.Spec.Components, v1beta1.DynamoComponentDeploymentSharedSpec{ + ComponentName: frontend, ComponentType: v1beta1.ComponentTypeFrontend, + }) + } + return dgd +} + +// newPoolTestRelay returns the DynamoPoolRelay that makes dgd a pool of cluster +// east. +func newPoolTestRelay(dgd *v1beta1.DynamoGraphDeployment) *v1alpha1.DynamoPoolRelay { + return &v1alpha1.DynamoPoolRelay{ + Name: dgd.Name, Namespace: dgd.Namespace, + Spec: v1alpha1.DynamoPoolRelaySpec{ClusterID: "east"}, + } +} + +// ownedServiceExport returns the export of the Service named name that dgd +// controls. +func ownedServiceExport( + t *testing.T, + scheme *runtime.Scheme, + dgd *v1beta1.DynamoGraphDeployment, + name string, +) *unstructured.Unstructured { + t.Helper() + export := multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name) + if err := controllerutil.SetControllerReference(dgd, export, scheme); err != nil { + t.Fatalf("set the owner of ServiceExport %s: %v", name, err) + } + return export +} + +// withExportConditions sets the conditions an MCS implementation writes on export. +func withExportConditions(export *unstructured.Unstructured, conditions ...metav1.Condition) *unstructured.Unstructured { + items := make([]any, 0, len(conditions)) + for _, condition := range conditions { + items = append(items, map[string]any{ + "type": condition.Type, "status": string(condition.Status), + "reason": condition.Reason, "message": condition.Message, + }) + } + export.Object["status"] = map[string]any{"conditions": items} + return export +} + +// imported is the condition a standard implementation writes on an export it has +// imported. +var imported = metav1.Condition{Type: "Ready", Status: metav1.ConditionTrue, Reason: "Exported"} + +// newServiceExportTestReconciler returns the export reconciler with the gate +// that implementation resolves to, reading exports through c, in a cluster with +// clusterSetID as its ID in its ClusterSet. +func newServiceExportTestReconciler( + c client.Client, + recorder events.EventRecorder, + implementation configv1alpha1.MCSImplementation, + clusterSetID string, +) *dgdServiceExportReconciler { + config := &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: implementation}, + } + gates := features.Gates{MultiClusterExport: implementation != configv1alpha1.MCSImplementationNone} + return newDGDServiceExportReconciler(newDGDResourceSyncer(c, recorder), config, gates, c, + func(context.Context) string { return clusterSetID }) +} + func TestDGDServiceExportReconciler(t *testing.T) { - const namespace = "dynamo" // -pool-relay is 56 characters, one more than Karmada imports; // -frontend is 54. longName := strings.Repeat("a", 45) for _, tc := range []struct { - name string - dgdName string - pool bool - implementation configv1alpha1.MCSImplementation - ownedBefore []string - terminating bool - foreign []string + name string + dgdName string + frontends []string + relay bool + relayTerminating bool + implementation configv1alpha1.MCSImplementation + ownedBefore []string + terminating bool + foreign []string + // ownedByAnother are exports another DGD controls. + ownedByAnother []string wantOwned []string wantEvents int - // Warnings that every reconcile repeats. - repeatedEvents int + // wantReason is the condition's reason once the exports are in place, or + // empty when the DGD gets no condition. The fake client runs no MCS + // implementation, so a standard implementation never reports an import. + wantReason string }{ { - name: "a pool exports its PoolRelay and frontend Services", - dgdName: "qwen", - pool: true, + name: "a pool exports its PoolRelay and frontend Services", + dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"qwen-pool-relay", "qwen-frontend"}, + wantEvents: 2, + wantReason: reasonPoolExported, + }, + { + name: "a standard implementation is waited for", + dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, implementation: configv1alpha1.MCSImplementationStandard, wantOwned: []string{"qwen-pool-relay", "qwen-frontend"}, wantEvents: 2, + wantReason: reasonPoolExportPending, + }, + { + name: "each frontend component's Service is exported under its own name", + dgdName: "qwen", frontends: []string{"Frontend", "Web.v2"}, relay: true, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"qwen-pool-relay", "qwen-frontend", "qwen-web-v2"}, + wantEvents: 3, + wantReason: reasonPoolExported, }, { - name: "exports use the Service names, with dots replaced", - dgdName: "qwen3-0.6b", - pool: true, + name: "a renamed frontend's export moves to its new Service", + dgdName: "qwen", frontends: []string{"Web"}, relay: true, implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{"qwen3-0-6b-pool-relay", "qwen3-0-6b-frontend"}, + ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, + wantOwned: []string{"qwen-pool-relay", "qwen-web"}, wantEvents: 2, + wantReason: reasonPoolExported, }, { - name: "removing the pool mark withdraws the exports", - dgdName: "qwen", + name: "a pool without a frontend exports its PoolRelay Service and reports it", + dgdName: "qwen", relay: true, + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{"qwen-pool-relay"}, + wantEvents: 1, + wantReason: reasonPoolNoFrontend, + }, + { + name: "a DGD with a dot in its name has no PoolRelay Service to export", + dgdName: "qwen3-0.6b", frontends: []string{"Frontend"}, relay: true, implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"qwen3-0-6b-frontend"}, + wantEvents: 1, + wantReason: reasonPoolInvalidServiceName, + }, + { + name: "deleting the DynamoPoolRelay withdraws the exports", + dgdName: "qwen", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, + wantEvents: 2, + }, + { + name: "a DynamoPoolRelay being deleted withdraws the exports", + dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, relayTerminating: true, + implementation: configv1alpha1.MCSImplementationStandard, ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, wantEvents: 2, }, { - name: "an export already being deleted is not deleted again", - dgdName: "qwen", + name: "an export already being deleted is not deleted again", + dgdName: "qwen", frontends: []string{"Frontend"}, implementation: configv1alpha1.MCSImplementationStandard, ownedBefore: []string{"qwen-pool-relay"}, terminating: true, wantOwned: []string{"qwen-pool-relay"}, }, { - name: "an export a user created is neither adopted nor deleted", - dgdName: "qwen", - pool: true, - implementation: configv1alpha1.MCSImplementationStandard, + name: "an export a user created is neither adopted nor deleted", + dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, + implementation: configv1alpha1.MCSImplementationKarmada, foreign: []string{"qwen-frontend"}, wantOwned: []string{"qwen-pool-relay"}, wantEvents: 1, + wantReason: reasonPoolExported, }, { - name: "a DGD that is not a pool leaves a user's export alone", - dgdName: "qwen", + name: "a DGD that is not a pool leaves a user's export alone", + dgdName: "qwen", frontends: []string{"Frontend"}, implementation: configv1alpha1.MCSImplementationStandard, foreign: []string{"qwen-pool-relay"}, }, { - name: "without an MCS implementation nothing is exported", - dgdName: "qwen", - pool: true, + name: "a DGD that is not a pool leaves another DGD's export alone", + dgdName: "qwen", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + ownedByAnother: []string{"qwen-frontend"}, + }, + { + name: "without an MCS implementation nothing is exported", + dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, implementation: configv1alpha1.MCSImplementationNone, }, { - name: "Karmada skips a Service name it cannot import", - dgdName: longName, - pool: true, + name: "Karmada skips a Service name it cannot import", + dgdName: longName, frontends: []string{"Frontend"}, relay: true, implementation: configv1alpha1.MCSImplementationKarmada, wantOwned: []string{longName + "-frontend"}, - wantEvents: 2, - repeatedEvents: 1, + wantEvents: 1, + wantReason: reasonPoolServiceNameTooLong, }, { - name: "standard MCS exports the same long names", - dgdName: longName, - pool: true, + name: "standard MCS exports the same long names", + dgdName: longName, frontends: []string{"Frontend"}, relay: true, implementation: configv1alpha1.MCSImplementationStandard, wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, wantEvents: 2, + wantReason: reasonPoolExportPending, }, } { t.Run(tc.name, func(t *testing.T) { g := gomega.NewGomegaWithT(t) ctx := context.Background() - t.Log("Create the DGD and the exports that exist before reconciling") + t.Log("Create the DGD, its DynamoPoolRelay if it is a pool, and the exports that exist before reconciling") scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := &v1beta1.DynamoGraphDeployment{ - Name: tc.dgdName, Namespace: namespace, UID: uuid.NewUUID(), - } - if tc.pool { - dgd.Annotations = map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"} - } + dgd := newPoolTestDGD(tc.dgdName, tc.frontends...) objects := []client.Object{dgd} + if tc.relay { + relay := newPoolTestRelay(dgd) + if tc.relayTerminating { + relay.Finalizers = []string{"example.com/hold"} + relay.DeletionTimestamp = &metav1.Time{Time: time.Now()} + } + objects = append(objects, relay) + } for _, name := range tc.ownedBefore { - owned := multicluster.RenderServiceExport(namespace, name, dgd.Name) - g.Expect(controllerutil.SetControllerReference(dgd, owned, scheme)).To(gomega.Succeed()) + owned := ownedServiceExport(t, scheme, dgd, name) if tc.terminating { owned.SetFinalizers([]string{"example.com/hold"}) - owned.SetDeletionTimestamp(&metav1.Time{Time: metav1.Now().Time}) + owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) } objects = append(objects, owned) } for _, name := range tc.foreign { - objects = append(objects, multicluster.RenderServiceExport(namespace, name, dgd.Name)) + objects = append(objects, multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name)) + } + another := newPoolTestDGD("another", "Frontend") + for _, name := range tc.ownedByAnother { + objects = append(objects, ownedServiceExport(t, scheme, another, name)) } fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, objects...) t.Log("Reconcile the exports, with the gate the configured implementation resolves to") - config := &configv1alpha1.OperatorConfiguration{ - MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: tc.implementation}, - } - gates := features.Gates{ - MultiClusterExport: tc.implementation != configv1alpha1.MCSImplementationNone, - } recorder := events.NewFakeRecorder(10) - reconciler := newDGDServiceExportReconciler(newDGDResourceSyncer(fakeClient, recorder), config, gates) - g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) + reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, "") + _, err := reconciler.Reconcile(ctx, dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) + + t.Log("Reconcile again: every export is in place, so nothing more is recorded, and the condition settles") + condition, err := reconciler.Reconcile(ctx, dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) + if tc.wantReason == "" { + g.Expect(condition).To(gomega.BeNil()) + } else { + g.Expect(condition).NotTo(gomega.BeNil()) + g.Expect(condition.Type).To(gomega.Equal(globalRouterPoolExportedConditionType)) + g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason), condition.Message) + } - t.Log("Reconcile again: every export is in place, so only warnings repeat") - g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) - g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents + tc.repeatedEvents)) - - t.Log("Check which exports the DGD controls, and that a user's export is untouched") - for _, name := range multicluster.ServiceNames(dgd.Name) { - export := multicluster.ServiceExportMetadata() - err := fakeClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, export) - switch { - case slices.Contains(tc.wantOwned, name): - g.Expect(err).NotTo(gomega.HaveOccurred(), name) - g.Expect(metav1.IsControlledBy(export, dgd)).To(gomega.BeTrue(), name) - g.Expect(export.Labels).To(gomega.HaveKeyWithValue(consts.KubeLabelDynamoGraphDeploymentName, dgd.Name)) - case slices.Contains(tc.foreign, name): - g.Expect(err).NotTo(gomega.HaveOccurred(), name) - g.Expect(metav1.IsControlledBy(export, dgd)).To(gomega.BeFalse(), name) - default: - g.Expect(apierrors.IsNotFound(err)).To(gomega.BeTrue(), "%s: %v", name, err) + t.Log("Check which exports the DGD controls, and that other exports are untouched") + var owned, others []string + for i, export := range listServiceExports(t, fakeClient, dgd.Namespace) { + if !metav1.IsControlledBy(&export, dgd) { + others = append(others, export.GetName()) + continue } + owned = append(owned, export.GetName()) + g.Expect(export.GetLabels()).To(gomega.HaveKeyWithValue(consts.KubeLabelDynamoGraphDeploymentName, dgd.Name), i) + } + g.Expect(owned).To(gomega.ConsistOf(tc.wantOwned)) + g.Expect(others).To(gomega.ConsistOf(append(append([]string{}, tc.foreign...), tc.ownedByAnother...))) + }) + } +} + +func TestDGDServiceExportReconcilerReportsTheImport(t *testing.T) { + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + relayConditions []metav1.Condition + frontendTerminating bool + wantStatus metav1.ConditionStatus + wantReason string + wantMessage string + }{ + { + name: "Karmada reports nothing", + implementation: configv1alpha1.MCSImplementationKarmada, + wantStatus: metav1.ConditionTrue, + wantReason: reasonPoolExported, + wantMessage: "Exported qwen-pool-relay, qwen-frontend through karmada", + }, + { + name: "a standard implementation imported both", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{imported}, + wantStatus: metav1.ConditionTrue, + wantReason: reasonPoolExported, + wantMessage: "Exported qwen-pool-relay, qwen-frontend through mcs", + }, + { + name: "a standard implementation has not reported yet", + implementation: configv1alpha1.MCSImplementationStandard, + wantStatus: metav1.ConditionUnknown, + wantReason: reasonPoolExportPending, + wantMessage: "Waiting for mcs to import qwen-pool-relay, qwen-frontend", + }, + { + name: "the MCS implementation finds no Service to export", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{ + {Type: "Valid", Status: metav1.ConditionFalse, Reason: "NoService", Message: "Service not found"}, + }, + wantStatus: metav1.ConditionFalse, + wantReason: reasonPoolExportRejected, + wantMessage: "mcs reports on the export of Service qwen-pool-relay: Valid is False (NoService): Service not found", + }, + { + name: "the import is pending", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{ + {Type: "Valid", Status: metav1.ConditionTrue}, {Type: "Ready", Status: metav1.ConditionFalse, Reason: "Pending"}, + }, + wantStatus: metav1.ConditionUnknown, + wantReason: reasonPoolExportPending, + wantMessage: "Waiting for mcs to import qwen-pool-relay", + }, + { + name: "the import failed", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{ + {Type: "Ready", Status: metav1.ConditionFalse, Reason: "Failed", Message: "refused"}, + }, + wantStatus: metav1.ConditionFalse, + wantReason: reasonPoolExportRejected, + wantMessage: "mcs reports on the export of Service qwen-pool-relay: Ready is False (Failed): refused", + }, + { + name: "the implementation resolved a conflict", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{ + imported, {Type: "Conflict", Status: metav1.ConditionTrue, Reason: "PortConflict", Message: "ports differ"}, + }, + wantStatus: metav1.ConditionTrue, + wantReason: reasonPoolExportConflict, + wantMessage: "Exported qwen-pool-relay, qwen-frontend through mcs, which resolves a conflict with another " + + "cluster's export of qwen-pool-relay (Conflict is True (PortConflict): ports differ)", + }, + { + name: "an export waits to be replaced", + implementation: configv1alpha1.MCSImplementationStandard, + relayConditions: []metav1.Condition{imported}, + frontendTerminating: true, + wantStatus: metav1.ConditionUnknown, + wantReason: reasonPoolExportPending, + wantMessage: "Waiting for mcs to import qwen-frontend", + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Create a pool whose exports exist, with what the MCS implementation reported on them") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := newPoolTestDGD("qwen", "Frontend") + relayExport := ownedServiceExport(t, scheme, dgd, "qwen-pool-relay") + frontendExport := ownedServiceExport(t, scheme, dgd, "qwen-frontend") + if len(tc.relayConditions) > 0 { + withExportConditions(relayExport, tc.relayConditions...) + withExportConditions(frontendExport, imported) + } + if tc.frontendTerminating { + frontendExport.SetFinalizers([]string{"example.com/hold"}) + frontendExport.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) } + fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, + dgd, newPoolTestRelay(dgd), relayExport, frontendExport) + + t.Log("Reconcile and read the pool's condition") + recorder := events.NewFakeRecorder(10) + reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, "") + condition, err := reconciler.Reconcile(context.Background(), dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(recorder.Events).To(gomega.BeEmpty()) + g.Expect(condition.Status).To(gomega.Equal(tc.wantStatus)) + g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason)) + g.Expect(condition.Message).To(gomega.Equal(tc.wantMessage)) }) } } +func TestDGDServiceExportReconcilerChecksTheClusterID(t *testing.T) { + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + // clusterSetID is the cluster's ID in its ClusterSet, or empty when unknown. + clusterSetID string + noClusterSetID bool + wantReason string + }{ + { + name: "a DynamoPoolRelay with the cluster's ID", + implementation: configv1alpha1.MCSImplementationStandard, + clusterSetID: "east", + wantReason: reasonPoolExported, + }, + { + name: "a DynamoPoolRelay with another cluster ID", + implementation: configv1alpha1.MCSImplementationStandard, + clusterSetID: "west", + wantReason: reasonPoolClusterIDMismatch, + }, + { + name: "a cluster whose ID in its ClusterSet is unknown", + implementation: configv1alpha1.MCSImplementationStandard, + wantReason: reasonPoolExported, + }, + { + name: "Karmada names member clusters itself", + implementation: configv1alpha1.MCSImplementationKarmada, + clusterSetID: "west", + wantReason: reasonPoolExported, + }, + { + name: "without a way to read the ID the check is skipped", + implementation: configv1alpha1.MCSImplementationStandard, + noClusterSetID: true, + wantReason: reasonPoolExported, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Create a pool of cluster east whose exports are imported") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := newPoolTestDGD("qwen", "Frontend") + fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, + dgd, newPoolTestRelay(dgd), + withExportConditions(ownedServiceExport(t, scheme, dgd, "qwen-pool-relay"), imported), + withExportConditions(ownedServiceExport(t, scheme, dgd, "qwen-frontend"), imported)) + + t.Log("Reconcile in a cluster with the given ID in its ClusterSet, and read the pool's condition") + reconciler := newServiceExportTestReconciler(fakeClient, events.NewFakeRecorder(10), tc.implementation, tc.clusterSetID) + if tc.noClusterSetID { + reconciler.clusterSetID = nil + } + condition, err := reconciler.Reconcile(context.Background(), dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason), condition.Message) + if tc.wantReason == reasonPoolClusterIDMismatch { + g.Expect(condition.Status).To(gomega.Equal(metav1.ConditionFalse)) + g.Expect(condition.Message).To(gomega.ContainSubstring(`clusterId "east"`)) + g.Expect(condition.Message).To(gomega.ContainSubstring(`the ID "west"`)) + } + }) + } +} + +func TestClusterSetIDs(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Serve the cluster's ID in its ClusterSet, counting reads and failing them on demand") + property := &unstructured.Unstructured{Object: map[string]any{"spec": map[string]any{"value": "east"}}} + property.SetGroupVersionKind(multicluster.ClusterPropertyGVK) + property.SetName(multicluster.ClusterIDProperty) + reads := 0 + var failure error + reader := newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + reads++ + if failure != nil { + return failure + } + return c.Get(ctx, key, obj, opts...) + }, + }, property) + ids := &clusterSetIDs{} + start := time.Now() + + t.Log("The first call reads the ID, and calls within the TTL reuse it") + g.Expect(ids.get(ctx, reader, start)).To(gomega.Equal("east")) + g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL-time.Second))).To(gomega.Equal("east")) + g.Expect(reads).To(gomega.Equal(1)) + + t.Log("After the TTL a failed read skips the check and is not kept") + failure = errors.New("connection refused") + g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL))).To(gomega.BeEmpty()) + failure = nil + g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL))).To(gomega.Equal("east")) + g.Expect(reads).To(gomega.Equal(3)) +} + func TestDGDServiceExportReconcilerToleratesAStaleCache(t *testing.T) { g := gomega.NewGomegaWithT(t) ctx := context.Background() t.Log("The cache has not yet seen exports that the API server already has") scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := &v1beta1.DynamoGraphDeployment{ - Name: "qwen", Namespace: "dynamo", UID: uuid.NewUUID(), - Annotations: map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"}, - } + dgd := newPoolTestDGD("qwen", "Frontend") fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{ Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { return apierrors.NewAlreadyExists(multicluster.ServiceExportGVK.GroupVersion().WithResource("serviceexports").GroupResource(), obj.GetName()) }, - }, dgd) + }, dgd, newPoolTestRelay(dgd)) - t.Log("Reconcile: AlreadyExists is no error, and nothing is announced") - config := &configv1alpha1.OperatorConfiguration{ - MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: configv1alpha1.MCSImplementationStandard}, - } + t.Log("Reconcile: AlreadyExists is no error, nothing is announced, and the exports are pending") recorder := events.NewFakeRecorder(10) - reconciler := newDGDServiceExportReconciler( - newDGDResourceSyncer(fakeClient, recorder), - config, - features.Gates{MultiClusterExport: true}, - ) - g.Expect(reconciler.Reconcile(ctx, dgd)).To(gomega.Succeed()) + reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, "") + condition, err := reconciler.Reconcile(ctx, dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(recorder.Events).To(gomega.BeEmpty()) + g.Expect(condition.Reason).To(gomega.Equal(reasonPoolExportPending)) } func TestMapServiceExportToDGDRequests(t *testing.T) { @@ -256,26 +609,162 @@ func TestMapServiceExportToDGDRequests(t *testing.T) { t.Log("Create two DGDs in one namespace, one with a dot in its name, and a third elsewhere") scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dotted := &v1beta1.DynamoGraphDeployment{Name: "qwen3-0.6b", Namespace: "dynamo"} - llama := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "dynamo"} - elsewhere := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "other"} + llama := newPoolTestDGD("llama", "Frontend", "Web") + dotted := newPoolTestDGD("qwen3-0.6b", "Frontend") + elsewhere := newPoolTestDGD("llama", "Frontend") + elsewhere.Namespace = "other" reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, interceptor.Funcs{}, dotted, llama, elsewhere), + Client: newServiceExportTestClient(scheme, interceptor.Funcs{}, llama, dotted, elsewhere), } for _, tc := range []struct { - export string + name string + export *unstructured.Unstructured want []ctrl.Request }{ - {export: "qwen3-0-6b-pool-relay", want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(dotted)}}}, - {export: "llama-frontend", want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}}, - {export: "llama", want: []ctrl.Request{}}, + { + name: "a PoolRelay Service", + export: multicluster.RenderServiceExport("dynamo", "llama-pool-relay", "llama"), + want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, + }, + { + name: "a frontend Service under its component's name", + export: multicluster.RenderServiceExport("dynamo", "llama-web", "llama"), + want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, + }, + { + name: "the frontend Service of a DGD with a dot in its name", + export: multicluster.RenderServiceExport("dynamo", "qwen3-0-6b-frontend", "qwen3-0.6b"), + want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(dotted)}}, + }, + { + name: "an export the DGD controls for a frontend it no longer has", + export: ownedServiceExport(t, scheme, llama, "llama-old"), + want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, + }, + { + name: "another Service", + export: multicluster.RenderServiceExport("dynamo", "llama", "llama"), + want: []ctrl.Request{}, + }, } { - t.Log("Map an event for export " + tc.export + " in namespace dynamo to the DGDs that export it") - export := multicluster.ServiceExportMetadata() - export.Namespace = "dynamo" - export.Name = tc.export - g.Expect(reconciler.mapServiceExportToDGDRequests(ctx, export)).To(gomega.Equal(tc.want), tc.export) + t.Log("Map an event for " + tc.name + " in namespace dynamo to the DGDs it concerns") + g.Expect(reconciler.mapServiceExportToDGDRequests(ctx, tc.export)).To(gomega.Equal(tc.want), tc.name) + } +} + +func TestPoolWatchPredicates(t *testing.T) { + g := gomega.NewGomegaWithT(t) + deleting := &metav1.Time{Time: time.Now()} + + t.Log("A DynamoPoolRelay event requeues the DGD of the same name") + relay := newPoolTestRelay(newPoolTestDGD("qwen")) + g.Expect(mapPoolRelayToDGDRequests(context.Background(), relay)). + To(gomega.Equal([]ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(relay)}})) + + t.Log("A DynamoPoolRelay update counts when its spec changes or its deletion starts") + respecified := relay.DeepCopy() + respecified.Generation = 2 + relabeled := relay.DeepCopy() + relabeled.Labels = map[string]string{"team": "router"} + withdrawn := relay.DeepCopy() + withdrawn.DeletionTimestamp = deleting + g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: respecified})).To(gomega.BeTrue()) + g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: withdrawn})).To(gomega.BeTrue()) + g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: relabeled})).To(gomega.BeFalse()) + + t.Log("A ServiceExport update counts when the MCS implementation reports or its deletion starts") + export := multicluster.RenderServiceExport("dynamo", "qwen-frontend", "qwen") + reported := withExportConditions(export.DeepCopy(), imported) + relabeledExport := export.DeepCopy() + relabeledExport.SetLabels(map[string]string{"team": "router"}) + deletingExport := export.DeepCopy() + deletingExport.SetDeletionTimestamp(deleting) + g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: reported})).To(gomega.BeTrue()) + g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: deletingExport})).To(gomega.BeTrue()) + g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: relabeledExport})).To(gomega.BeFalse()) + g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: &corev1.Service{}, ObjectNew: &corev1.Service{}})).To(gomega.BeFalse()) +} + +func TestApplyPoolExportCondition(t *testing.T) { + condition := func(status metav1.ConditionStatus, reason string) *metav1.Condition { + return &metav1.Condition{ + Type: globalRouterPoolExportedConditionType, Status: status, Reason: reason, Message: reason + " message", + } + } + for _, tc := range []struct { + name string + previous *metav1.Condition + condition *metav1.Condition + wantEvents []string + }{ + { + name: "a new problem is set and announced", + condition: condition(metav1.ConditionFalse, reasonPoolNoFrontend), + wantEvents: []string{reasonPoolNoFrontend}, + }, + { + name: "a problem already reported is not announced again", + previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), + condition: condition(metav1.ConditionFalse, reasonPoolNoFrontend), + }, + { + name: "another problem is announced", + previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), + condition: condition(metav1.ConditionFalse, reasonPoolClusterIDMismatch), + wantEvents: []string{reasonPoolClusterIDMismatch}, + }, + { + name: "a conflict the implementation resolved is announced", + previous: condition(metav1.ConditionTrue, reasonPoolExported), + condition: condition(metav1.ConditionTrue, reasonPoolExportConflict), + wantEvents: []string{reasonPoolExportConflict}, + }, + { + name: "a failed reconcile records its own event", + condition: condition(metav1.ConditionFalse, reasonPoolExportFailed), + }, + { + name: "a pending import is not announced", + condition: condition(metav1.ConditionUnknown, reasonPoolExportPending), + }, + { + name: "an export that works is not announced", + previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), + condition: condition(metav1.ConditionTrue, reasonPoolExported), + }, + { + name: "a DGD that is no longer a pool loses the condition", + previous: condition(metav1.ConditionTrue, reasonPoolExported), + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Start from the status the DGD has, as the workload program does") + dgd := newPoolTestDGD("qwen", "Frontend") + if tc.previous != nil { + meta.SetStatusCondition(&dgd.Status.Conditions, *tc.previous) + } + result := newWorkloadProgramResult(dgd) + + t.Log("Apply the pool's condition") + applyPoolExportCondition(dgd, &result, tc.condition) + applied := meta.FindStatusCondition(result.Status.Conditions, globalRouterPoolExportedConditionType) + if tc.condition == nil { + g.Expect(applied).To(gomega.BeNil()) + } else { + g.Expect(applied).NotTo(gomega.BeNil()) + g.Expect(applied.Status).To(gomega.Equal(tc.condition.Status)) + g.Expect(applied.Reason).To(gomega.Equal(tc.condition.Reason)) + } + reasons := make([]string, 0, len(result.Events)) + for _, recorded := range result.Events { + g.Expect(recorded.Type).To(gomega.Equal(corev1.EventTypeWarning)) + reasons = append(reasons, recorded.Reason) + } + g.Expect(reasons).To(gomega.ConsistOf(tc.wantEvents)) + }) } } @@ -283,10 +772,10 @@ func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { g := gomega.NewGomegaWithT(t) ctx := context.Background() - t.Log("Mark a DGD as a pool, and refuse every ServiceExport create, as an admission policy would") + t.Log("Make a DGD a pool, and refuse every ServiceExport create, as an admission policy would") scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := &v1beta1.DynamoGraphDeployment{Name: "llama", Namespace: "dynamo", UID: uuid.NewUUID()} - dgd.Annotations = map[string]string{consts.KubeAnnotationGlobalRouterPool: "true"} + dgd := newPoolTestDGD("llama", "Frontend") + relay := newPoolTestRelay(dgd) refused := interceptor.Funcs{ Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { return apierrors.NewForbidden( @@ -296,24 +785,169 @@ func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { } recorder := events.NewFakeRecorder(10) reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, refused, dgd), + Client: newServiceExportTestClient(scheme, refused, dgd, relay), Recorder: recorder, Config: &configv1alpha1.OperatorConfiguration{ MultiCluster: configv1alpha1.MultiClusterConfiguration{ - MCSImplementation: configv1alpha1.MCSImplementationStandard, + MCSImplementation: configv1alpha1.MCSImplementationKarmada, }, }, RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: features.Gates{MultiClusterExport: true}}, } - t.Log("The failure asks for a retry and is recorded as a warning, without an error that would fail the DGD") - g.Expect(reconciler.reconcilePoolExports(ctx, dgd)).To(gomega.BeTrue()) + t.Log("The failure asks for a retry, is recorded as a warning and in the condition, and returns no error that would fail the DGD") + result := reconciler.reconcilePoolExports(ctx, dgd) + g.Expect(result.retry).To(gomega.BeTrue()) + g.Expect(result.condition.Status).To(gomega.Equal(metav1.ConditionFalse)) + g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExportFailed)) g.Expect(recorder.Events).To(gomega.HaveLen(1)) g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) + t.Log("A failure to read the DynamoPoolRelay leaves the condition as it was, since the DGD may not be a pool") + meta.SetStatusCondition(&dgd.Status.Conditions, *newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExported, "exported")) + reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*v1alpha1.DynamoPoolRelay); ok { + return errors.New("cache not synced") + } + return c.Get(ctx, key, obj, opts...) + }, + }, dgd, relay) + result = reconciler.reconcilePoolExports(ctx, dgd) + g.Expect(result.retry).To(gomega.BeTrue()) + g.Expect(result.condition).NotTo(gomega.BeNil()) + g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExported)) + g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) + + t.Log("A failed withdrawal from a DGD that is no longer a pool keeps the condition until the exports are gone") + former := newPoolTestDGD("qwen", "Frontend") + meta.SetStatusCondition(&former.Status.Conditions, *newPoolCondition(former, metav1.ConditionTrue, reasonPoolExported, "exported")) + reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return errors.New("denied by policy") + }, + }, former, ownedServiceExport(t, scheme, former, "qwen-frontend")) + result = reconciler.reconcilePoolExports(ctx, former) + g.Expect(result.retry).To(gomega.BeTrue()) + g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExported)) + g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) + t.Log("Exports that succeed ask for no retry") - reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{}, dgd) - g.Expect(reconciler.reconcilePoolExports(ctx, dgd)).To(gomega.BeFalse()) + reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{}, dgd, relay) + result = reconciler.reconcilePoolExports(ctx, dgd) + g.Expect(result.retry).To(gomega.BeFalse()) + g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExportPending)) +} + +// newPoolTestDGDReconciler returns a DGD reconciler over objects, with standard +// MCS and the gates given. +func newPoolTestDGDReconciler( + t *testing.T, + gates features.Gates, + objects ...client.Object, +) (*DynamoGraphDeploymentReconciler, *events.FakeRecorder) { + gates.MultiClusterExport = true + recorder := events.NewFakeRecorder(10) + return &DynamoGraphDeploymentReconciler{ + Client: newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), interceptor.Funcs{}, objects...), + Recorder: recorder, + Config: &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{ + MCSImplementation: configv1alpha1.MCSImplementationStandard, + }, + }, + RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: gates}, + DockerSecretRetriever: &mockDockerSecretRetriever{ + GetSecretsFunc: func(namespace, imageName string) ([]string, error) { return nil, nil }, + }, + }, recorder +} + +// drainEvents returns the events recorder holds. +func drainEvents(recorder *events.FakeRecorder) []string { + var recorded []string + for len(recorder.Events) > 0 { + recorded = append(recorded, <-recorder.Events) + } + return recorded +} + +func TestDynamoGraphDeploymentReconcilePersistsThePoolCondition(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := t.Context() + + t.Log("Create a finalized DGD on the component provider, whose program rejects its only component, and its DynamoPoolRelay") + dgd := &v1beta1.DynamoGraphDeployment{ + Name: "llama", Namespace: "dynamo", Generation: 4, UID: uuid.NewUUID(), + Annotations: map[string]string{consts.KubeAnnotationWorkloadProvider: consts.WorkloadProviderComponent}, + Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + {ComponentName: "serving", ComponentType: v1beta1.ComponentTypeLPX}, + }}, + } + commoncontroller.AddFinalizer(dgd) + relay := newPoolTestRelay(dgd) + reconciler, recorder := newPoolTestDGDReconciler(t, features.Gates{LPX: true}, dgd, relay) + reconcileAndRead := func() (*v1beta1.DynamoGraphDeployment, []string) { + _, err := reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) + g.Expect(errors.Is(err, reconcile.TerminalError(nil))).To(gomega.BeTrue(), "%v", err) + stored := &v1beta1.DynamoGraphDeployment{} + g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) + return stored, drainEvents(recorder) + } + noFrontend := gomega.HavePrefix(corev1.EventTypeWarning + " " + reasonPoolNoFrontend) + + t.Log("Reconcile: the program's status is persisted with the pool's condition, and the problem is announced") + stored, recorded := reconcileAndRead() + ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready") + g.Expect(ready).NotTo(gomega.BeNil()) + g.Expect(ready.Reason).To(gomega.Equal("UnsupportedComponent")) + pool := meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType) + g.Expect(pool).NotTo(gomega.BeNil()) + g.Expect(pool.Status).To(gomega.Equal(metav1.ConditionFalse)) + g.Expect(pool.Reason).To(gomega.Equal(reasonPoolNoFrontend)) + g.Expect(pool.ObservedGeneration).To(gomega.Equal(dgd.Generation)) + g.Expect(recorded).To(gomega.ContainElement(noFrontend)) + + t.Log("Reconcile again: the same problem is not announced twice") + _, recorded = reconcileAndRead() + g.Expect(recorded).NotTo(gomega.ContainElement(noFrontend)) + + t.Log("Delete the DynamoPoolRelay: the export is withdrawn and the condition removed") + g.Expect(reconciler.Delete(ctx, relay)).To(gomega.Succeed()) + stored, _ = reconcileAndRead() + g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType)).To(gomega.BeNil()) + g.Expect(listServiceExports(t, reconciler.Client, dgd.Namespace)).To(gomega.BeEmpty()) +} + +func TestDynamoGraphDeploymentReconcileKeepsThePoolConditionOnEarlyFailures(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := t.Context() + + t.Log("Store a DGD with an LPX component while LPX is disabled, and its DynamoPoolRelay") + dgd := &v1beta1.DynamoGraphDeployment{ + Name: "mixed", Namespace: "dynamo", Generation: 4, UID: uuid.NewUUID(), + Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + {ComponentName: "worker", ComponentType: v1beta1.ComponentTypeLPX}, + }}, + } + relay := newPoolTestRelay(dgd) + reconciler, _ := newPoolTestDGDReconciler(t, features.Gates{}, dgd, relay) + + t.Log("Reconcile: the early failure's status carries the pool's condition") + _, err := reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) + g.Expect(err).NotTo(gomega.HaveOccurred()) + stored := &v1beta1.DynamoGraphDeployment{} + g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) + g.Expect(meta.FindStatusCondition(stored.Status.Conditions, "Ready").Reason).To(gomega.Equal("lpx_disabled")) + g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType).Reason). + To(gomega.Equal(reasonPoolNoFrontend)) + + t.Log("Delete the DynamoPoolRelay: the next early failure removes the condition") + g.Expect(reconciler.Delete(ctx, relay)).To(gomega.Succeed()) + _, err = reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) + g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType)).To(gomega.BeNil()) } func TestRetryPoolExports(t *testing.T) { diff --git a/deploy/operator/internal/controller/dgd_workload_program_test.go b/deploy/operator/internal/controller/dgd_workload_program_test.go index e8e71674e41a..5470d24248a1 100644 --- a/deploy/operator/internal/controller/dgd_workload_program_test.go +++ b/deploy/operator/internal/controller/dgd_workload_program_test.go @@ -256,7 +256,7 @@ func TestPersistWorkloadProgramResultEmitsEventsAfterStatusUpdate(t *testing.T) result.Eventf(corev1.EventTypeNormal, "Transition", "transition persisted") t.Log("Persist status through the outer controller boundary") - err := reconciler.persistWorkloadProgramResult(context.Background(), dgd, result) + err := reconciler.persistWorkloadProgramResult(context.Background(), dgd, result, nil) t.Log("Verify event publication is strictly ordered after successful status persistence") require.True(t, statusUpdated) diff --git a/deploy/operator/internal/controller/dynamographdeployment_controller.go b/deploy/operator/internal/controller/dynamographdeployment_controller.go index 2422752a7792..97d9dd4075b6 100644 --- a/deploy/operator/internal/controller/dynamographdeployment_controller.go +++ b/deploy/operator/internal/controller/dynamographdeployment_controller.go @@ -21,6 +21,7 @@ import ( "context" "errors" "fmt" + "slices" "strings" "time" @@ -30,6 +31,7 @@ import ( corev1 "k8s.io/api/core/v1" resourcev1 "k8s.io/api/resource/v1" "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/client-go/rest" "k8s.io/client-go/tools/events" @@ -78,6 +80,16 @@ type DynamoGraphDeploymentReconciler struct { Recorder events.EventRecorder DockerSecretRetriever DockerSecretRetriever SSHKeyManager *secret.SSHKeyManager + // APIReader reads objects that the cache does not hold, such as the cluster's + // ID in its ClusterSet. + APIReader client.Reader + // Cache reads unstructured objects, such as ServiceExports, from the + // informers that the controller's watches start. The manager's client reads + // unstructured objects from the API server. + Cache client.Reader + // clusterSetIDs keeps the cluster's ID in its ClusterSet between reconciles. + // Nil skips the check of a pool's cluster ID. + clusterSetIDs *clusterSetIDs } // +kubebuilder:rbac:groups=nvidia.com,resources=dynamographdeployments,verbs=get;list;watch;create;update;patch;delete @@ -95,6 +107,8 @@ type DynamoGraphDeploymentReconciler struct { // +kubebuilder:rbac:groups=inference.networking.k8s.io,resources=inferencepools,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=networking.istio.io,resources=destinationrules,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=multicluster.x-k8s.io,resources=serviceexports,verbs=get;list;watch;create;delete +// +kubebuilder:rbac:groups=nvidia.com,resources=dynamopoolrelays,verbs=get;list;watch +// +kubebuilder:rbac:groups=about.k8s.io,resources=clusterproperties,verbs=get // +kubebuilder:rbac:groups=nvidia.com,resources=snapshotjobs,verbs=get;list;watch;create;patch;delete // +kubebuilder:rbac:groups=nvidia.com,resources=podsnapshots,verbs=get;list;watch;patch;delete // +kubebuilder:rbac:groups=resource.k8s.io,resources=resourceclaimtemplates,verbs=get;list;watch;create;update;patch;delete @@ -135,10 +149,11 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, err } - // A Global Router pool's exports follow its annotation whatever happens to its - // workloads below. A failed export never fails the DGD: it is logged, recorded - // as an event, and retried. - if r.reconcilePoolExports(ctx, dynamoDeployment) { + // A Global Router pool's exports follow its DynamoPoolRelay whatever happens to + // its workloads below. A failed export never fails the DGD: it is logged, + // recorded as an event and in the pool's condition, and retried. + poolExports := r.reconcilePoolExports(ctx, dynamoDeployment) + if poolExports.retry { defer func() { result = retryPoolExports(result, err) }() } @@ -146,7 +161,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr if dynamoDeployment.HasLPXComponent() && !r.RuntimeConfig.Gate.Enabled(features.LPX) { programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, "lpx_disabled", errors.New("LPX integration is disabled")) - return programResult.Result, r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult) + return programResult.Result, r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition) } // Lock in the provider before allowing any other reconciliation effects. @@ -161,7 +176,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr // Persist the invalid selection diagnosis before suppressing automatic retries. programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, reasonUnsupportedWorkloadProvider, err) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { return programResult.Result, statusErr } return programResult.Result, reconcile.TerminalError(err) @@ -184,7 +199,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr if compatibilityErr := errors.Join(compatibilityErrs...); compatibilityErr != nil { programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, reasonFailedToReconcileResources, compatibilityErr) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { logger.Error(statusErr, "unable to update status after compatibility failure") return ctrl.Result{}, statusErr } @@ -196,7 +211,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr logger.Error(err, "failed to handle the finalizer") programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, "failed_to_handle_the_finalizer", err) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { logger.Error(statusErr, "unable to update status after finalizer failure") } return ctrl.Result{}, err @@ -222,7 +237,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr "Refusing to reconcile a resource with conflicting controller ownership: %s", ownershipConflictCondition.Message) } result = programResult.Result - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { logger.Error(statusErr, "unable to persist workload program status") return result, statusErr } @@ -235,11 +250,15 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr return result, nil } +// persistWorkloadProgramResult writes result's status, with the Global Router +// pool's condition, and records result's events once the status is written. func (r *DynamoGraphDeploymentReconciler) persistWorkloadProgramResult( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, result workloadProgramResult, + poolCondition *metav1.Condition, ) error { + applyPoolExportCondition(dgd, &result, poolCondition) dgd.Status = result.Status if err := r.Status().Update(ctx, dgd); err != nil { return fmt.Errorf("update DynamoGraphDeployment status: %w", err) @@ -278,24 +297,80 @@ func workloadRoutingAnnotationsChanged(update event.UpdateEvent) bool { // poolExportRetry is how soon a DGD whose pool exports failed is reconciled again. const poolExportRetry = 30 * time.Second -// reconcilePoolExports converges a Global Router pool's ServiceExports and reports -// whether that failed, so that the DGD is reconciled again. A failure is logged -// and recorded as a Warning event. +// poolExportResult is the outcome of reconciling a Global Router pool's exports. +type poolExportResult struct { + // retry asks for the DGD to be reconciled again because the exports failed. + retry bool + // condition is the GlobalRouterPoolExported condition for the DGD, or nil to + // remove it. + condition *metav1.Condition +} + +// reconcilePoolExports converges a Global Router pool's ServiceExports. A failure +// is logged, recorded as a Warning event and, when the DGD is known to be a +// pool, in its condition, and retried. func (r *DynamoGraphDeploymentReconciler) reconcilePoolExports( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, -) bool { - exports := newDGDServiceExportReconciler(newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate) - err := exports.Reconcile(ctx, dgd) +) poolExportResult { + exports := newDGDServiceExportReconciler( + newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate, + r.serviceExportReader(), r.clusterSetID) + condition, err := exports.Reconcile(ctx, dgd) if err == nil { - return false + return poolExportResult{condition: condition} } log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") if r.Recorder != nil { r.Recorder.Eventf(dgd, nil, corev1.EventTypeWarning, "PoolExportFailed", "Export", "Failed to reconcile Global Router pool ServiceExports: %v", err) } - return true + // Whether the DGD is a pool is unknown, so its condition stays as it is. + if condition == nil { + if previous := meta.FindStatusCondition(dgd.Status.Conditions, globalRouterPoolExportedConditionType); previous != nil { + kept := *previous + condition = &kept + } + } + return poolExportResult{retry: true, condition: condition} +} + +// serviceExportReader returns the reader of ServiceExports: the cache, or the +// client when the controller has no cache, as in tests. +func (r *DynamoGraphDeploymentReconciler) serviceExportReader() client.Reader { + if r.Cache != nil { + return r.Cache + } + return r.Client +} + +// clusterSetID returns the cluster's ID in its ClusterSet, or "" when it is +// unknown. +func (r *DynamoGraphDeploymentReconciler) clusterSetID(ctx context.Context) string { + if r.APIReader == nil || r.clusterSetIDs == nil { + return "" + } + return r.clusterSetIDs.get(ctx, r.APIReader, time.Now()) +} + +// applyPoolExportCondition sets the GlobalRouterPoolExported condition in the +// status about to be persisted, or removes it when condition is nil. A problem +// that the condition newly reports is also recorded as a Warning event. +func applyPoolExportCondition( + dgd *nvidiacomv1beta1.DynamoGraphDeployment, + result *workloadProgramResult, + condition *metav1.Condition, +) { + if condition == nil { + meta.RemoveStatusCondition(&result.Status.Conditions, globalRouterPoolExportedConditionType) + return + } + previous := meta.FindStatusCondition(dgd.Status.Conditions, globalRouterPoolExportedConditionType) + meta.SetStatusCondition(&result.Status.Conditions, *condition) + raised := previous == nil || previous.Status != condition.Status || previous.Reason != condition.Reason + if raised && slices.Contains(poolProblemReasons, condition.Reason) { + result.Eventf(corev1.EventTypeWarning, condition.Reason, "%s", condition.Message) + } } // retryPoolExports makes a reconcile run again within poolExportRetry, unless it @@ -429,15 +504,26 @@ func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) err })) } // A Global Router pool's exports converge once the cache sees them created or - // deleted, the DGD's own and a user's of the same Service alike. + // deleted, the DGD's own and a user's of the same Service alike, and its + // condition follows what the MCS implementation reports on them. A + // DynamoPoolRelay makes the DGD of the same name a pool. if r.RuntimeConfig.Gate.Enabled(features.MultiClusterExport) { ctrlBuilder = ctrlBuilder.Watches( - multicluster.ServiceExportMetadata(), + multicluster.NewServiceExport(), handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToDGDRequests), builder.WithPredicates(predicate.Funcs{ CreateFunc: func(ce event.CreateEvent) bool { return true }, DeleteFunc: func(de event.DeleteEvent) bool { return true }, - UpdateFunc: func(ue event.UpdateEvent) bool { return false }, + UpdateFunc: serviceExportStateChanged, + GenericFunc: func(ge event.GenericEvent) bool { return false }, + }), + ).Watches( + &nvidiacomv1alpha1.DynamoPoolRelay{}, + handler.EnqueueRequestsFromMapFunc(mapPoolRelayToDGDRequests), + builder.WithPredicates(predicate.Funcs{ + CreateFunc: func(ce event.CreateEvent) bool { return true }, + DeleteFunc: func(de event.DeleteEvent) bool { return true }, + UpdateFunc: poolRelayChanged, GenericFunc: func(ge event.GenericEvent) bool { return false }, }), ) diff --git a/deploy/operator/internal/controller/poolrelay/controller.go b/deploy/operator/internal/controller/poolrelay/controller.go index 0afd94c49c29..66cbdd0dc6b2 100644 --- a/deploy/operator/internal/controller/poolrelay/controller.go +++ b/deploy/operator/internal/controller/poolrelay/controller.go @@ -451,7 +451,7 @@ func sourceJSON(namespaces []string, connectionRevision string) string { return string(encoded) } -func serviceName(dgdName string) string { return dgdName + multicluster.PoolRelayServiceSuffix } +func serviceName(dgdName string) string { return multicluster.PoolRelayServiceName(dgdName) } func render(relay *v1alpha1.DynamoPoolRelay, namespaces []string) ([]client.Object, *appsv1.Deployment, error) { name := serviceName(relay.Name) diff --git a/deploy/operator/internal/controller/setup.go b/deploy/operator/internal/controller/setup.go index 56355c42473d..d3b586a3e941 100644 --- a/deploy/operator/internal/controller/setup.go +++ b/deploy/operator/internal/controller/setup.go @@ -102,6 +102,9 @@ func SetupDynamoGraphDeployment(mgr ctrl.Manager, opts DynamoGraphDeploymentSetu RestConfig: mgr.GetConfig(), DockerSecretRetriever: opts.DockerSecretRetriever, SSHKeyManager: opts.SSHKeyManager, + APIReader: mgr.GetAPIReader(), + Cache: mgr.GetCache(), + clusterSetIDs: &clusterSetIDs{}, }).SetupWithManager(mgr); err != nil { return fmt.Errorf("unable to create DynamoGraphDeployment controller: %w", err) } diff --git a/deploy/operator/internal/features/multicluster.go b/deploy/operator/internal/features/multicluster.go index 70069c48007c..a48ca93a35de 100644 --- a/deploy/operator/internal/features/multicluster.go +++ b/deploy/operator/internal/features/multicluster.go @@ -10,14 +10,14 @@ import ( ) // MultiClusterExport exports the PoolRelay and frontend Services of each DGD -// marked as a Global Router pool through the Multi-Cluster Services (MCS) API. +// that is a Global Router pool through the Multi-Cluster Services (MCS) API. // // Owner: @atchernych // Experimental since: v1.7.0 // Beta since: N/A // GA since: N/A -// Configuration: multiCluster.mcsImplementation, and the -// nvidia.com/global-router-pool annotation on each DGD +// Configuration: multiCluster.mcsImplementation, and a DynamoPoolRelay named +// after each pool's DGD // Auto-detection: N/A; API availability is verified when an implementation is configured // Requires: an MCS implementation, such as Karmada, serving // multicluster.x-k8s.io/v1alpha1 ServiceExport resources diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 17e05337ac6d..376d744cd01c 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -4,19 +4,29 @@ */ // Package multicluster renders the ServiceExports that announce a Global Router -// pool to other clusters through the Kubernetes Multi-Cluster Services (MCS) API. +// pool to other clusters through the Kubernetes Multi-Cluster Services (MCS) API, +// and reads what the MCS implementation reports about them. package multicluster import ( + "context" + "fmt" + "slices" "strings" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/validation" + "sigs.k8s.io/controller-runtime/pkg/client" ) // ServiceExportGVK is the SIG Multicluster ServiceExport. Every supported @@ -31,15 +41,24 @@ var ServiceExportGVK = schema.GroupVersionKind{ Kind: "ServiceExport", } +// ClusterPropertyGVK is the About API's ClusterProperty (KEP-2149), which holds +// facts about the cluster for its ClusterSet. +var ClusterPropertyGVK = schema.GroupVersionKind{ + Group: "about.k8s.io", + Version: "v1alpha1", + Kind: "ClusterProperty", +} + const ( - // PoolRelayServiceSuffix names a pool's PoolRelay Service, -pool-relay, - // as the Global Discovery Service expects by default. The operator creates this - // Service for a DynamoPoolRelay named after the DGD. - PoolRelayServiceSuffix = "-pool-relay" - // FrontendServiceSuffix names a pool's frontend Service, -frontend, as - // the Global Discovery Service expects by default. The operator creates this - // Service for a frontend component named "Frontend". - FrontendServiceSuffix = "-frontend" + // ClusterIDProperty names the ClusterProperty that holds the cluster's ID in its + // ClusterSet. A standard MCS implementation labels the EndpointSlices it imports + // with that ID, and the Global Discovery Service takes a pool's cluster from the + // label. + ClusterIDProperty = "cluster.clusterset.k8s.io" + + // poolRelayServiceSuffix names a pool's PoolRelay Service, as the Global + // Discovery Service expects by default. + poolRelayServiceSuffix = "-pool-relay" // maxKarmadaServiceNameLength is the longest Service name Karmada imports. // Karmada labels each collected slice kubernetes.io/service-name: @@ -47,21 +66,42 @@ const ( maxKarmadaServiceNameLength = 55 ) -// ServiceNames returns the names of a pool's PoolRelay and frontend Services, the -// Services the operator exports for a Global Router pool. Like the operator's own -// Service names, they replace the dots in a DGD name with dashes. -func ServiceNames(dgdName string) []string { - return []string{ - dynamo.NormalizeKubeResourceName(dgdName + PoolRelayServiceSuffix), - dynamo.NormalizeKubeResourceName(dgdName + FrontendServiceSuffix), +// PoolRelayServiceName returns the name of the PoolRelay Service of the DGD named +// dgdName, -pool-relay, as the Global Discovery Service expects by +// default. The operator creates this Service for a DynamoPoolRelay of the same +// name. +func PoolRelayServiceName(dgdName string) string { + return dgdName + poolRelayServiceSuffix +} + +// FrontendServiceNames returns the names of the Services the operator creates for +// the DGD's frontend components, in component order: -, +// lowercased, with dots replaced by dashes. +func FrontendServiceNames(dgd *v1beta1.DynamoGraphDeployment) []string { + var names []string + for i := range dgd.Spec.Components { + component := &dgd.Spec.Components[i] + if component.ComponentType != v1beta1.ComponentTypeFrontend { + continue + } + name := dynamo.NormalizeKubeResourceName(dynamo.GetDCDResourceName(dgd, component.ComponentName, "")) + if !slices.Contains(names, name) { + names = append(names, name) + } } + return names } -// IsPool reports whether a DGD with these annotations is marked as a Global Router -// pool: `true` in any case, as the DGD's other boolean annotations are read. A nil -// map is supported and is not a pool. -func IsPool(annotations map[string]string) bool { - return strings.EqualFold(annotations[commonconsts.KubeAnnotationGlobalRouterPool], "true") +// ServiceNames returns the Services the operator exports for the DGD as a Global +// Router pool: its PoolRelay Service, then its frontend Services. +func ServiceNames(dgd *v1beta1.DynamoGraphDeployment) []string { + names := []string{PoolRelayServiceName(dgd.Name)} + for _, name := range FrontendServiceNames(dgd) { + if !slices.Contains(names, name) { + names = append(names, name) + } + } + return names } // MaxServiceNameLength returns the longest Service name that implementation can @@ -74,10 +114,40 @@ func MaxServiceNameLength(implementation configv1alpha1.MCSImplementation) int { return validation.DNS1035LabelMaxLength } -// ServiceExportMetadata returns an empty metadata-only ServiceExport, to read, -// delete or watch exports through the cache without their content. -func ServiceExportMetadata() *metav1.PartialObjectMetadata { - serviceExport := &metav1.PartialObjectMetadata{} +// NameProblem is why a Service name cannot be exported. +type NameProblem int + +const ( + // NameExportable means that the implementation can export a Service of the name. + NameExportable NameProblem = iota + // NameInvalid means that no Service can have the name. + NameInvalid + // NameTooLong means that the implementation imports no Service name that long. + NameTooLong +) + +// CheckServiceName returns whether implementation can export a Service named name +// with the cluster it came from, and describes the problem when it cannot. +func CheckServiceName(implementation configv1alpha1.MCSImplementation, name string) (NameProblem, string) { + if problems := validation.IsDNS1035Label(name); len(problems) > 0 { + return NameInvalid, strings.Join(problems, "; ") + } + if maxLength := MaxServiceNameLength(implementation); len(name) > maxLength { + return NameTooLong, fmt.Sprintf("%s imports no Service name longer than %d characters", implementation, maxLength) + } + return NameExportable, "" +} + +// UsesClusterSetID reports whether implementation labels the EndpointSlices it +// imports with the cluster's ID in its ClusterSet, the ClusterIDProperty. Karmada +// names its member clusters itself. +func UsesClusterSetID(implementation configv1alpha1.MCSImplementation) bool { + return implementation == configv1alpha1.MCSImplementationStandard +} + +// NewServiceExport returns an empty ServiceExport, to watch exports or to name one. +func NewServiceExport() *unstructured.Unstructured { + serviceExport := &unstructured.Unstructured{} serviceExport.SetGroupVersionKind(ServiceExportGVK) return serviceExport } @@ -85,8 +155,7 @@ func ServiceExportMetadata() *metav1.PartialObjectMetadata { // RenderServiceExport returns the ServiceExport for the Service named name in // namespace, labeled with the DGD that exports it. It sets no owner reference. func RenderServiceExport(namespace, name, dgdName string) *unstructured.Unstructured { - serviceExport := &unstructured.Unstructured{} - serviceExport.SetGroupVersionKind(ServiceExportGVK) + serviceExport := NewServiceExport() serviceExport.SetNamespace(namespace) serviceExport.SetName(name) serviceExport.SetLabels(map[string]string{ @@ -94,3 +163,134 @@ func RenderServiceExport(namespace, name, dgdName string) *unstructured.Unstruct }) return serviceExport } + +// ExportState is what the MCS implementation reports about a ServiceExport. +type ExportState int + +const ( + // ExportPending means that the import is not finished, or that a standard + // implementation has not reported on the export yet. + ExportPending ExportState = iota + // ExportAccepted means that nothing is reported against the export. Karmada + // writes no conditions, so its exports are always accepted. + ExportAccepted + // ExportConflicting means that some property of the export conflicts with + // another cluster's export of the Service. The implementation resolves the + // conflict, so the import can still work. + ExportConflicting + // ExportRejected means that Valid is False, or Ready is False for a reason + // other than Pending. + ExportRejected +) + +// Export is a ServiceExport as the operator sees it. +type Export struct { + Name string + UID types.UID + // ControllerUID is the UID of the export's controller, empty without one. + ControllerUID types.UID + // Deleting is true once the export's deletion has started. + Deleting bool + // State is what the MCS implementation reports on the export, and Detail the + // condition that decided it, for people. + State ExportState + Detail string +} + +// ListExports returns the ServiceExports in namespace by name, with what +// implementation reports on them. reader should serve unstructured objects from +// a cache: the manager's client reads them from the API server. +func ListExports( + ctx context.Context, + reader client.Reader, + namespace string, + implementation configv1alpha1.MCSImplementation, +) (map[string]Export, error) { + list := &unstructured.UnstructuredList{} + list.SetGroupVersionKind(ServiceExportGVK.GroupVersion().WithKind(ServiceExportGVK.Kind + "List")) + if err := reader.List(ctx, list, client.InNamespace(namespace)); err != nil { + return nil, fmt.Errorf("failed to list ServiceExports: %w", err) + } + exports := make(map[string]Export, len(list.Items)) + for i := range list.Items { + item := &list.Items[i] + export := Export{Name: item.GetName(), UID: item.GetUID(), Deleting: item.GetDeletionTimestamp() != nil} + if controller := metav1.GetControllerOf(item); controller != nil { + export.ControllerUID = controller.UID + } + export.State, export.Detail = ReadExportState(implementation, item) + exports[export.Name] = export + } + return exports, nil +} + +// ReadExportState reads the conditions that implementation writes on a +// ServiceExport (KEP-1645: Valid, Ready and Conflict), and returns the export's +// state with the condition that decided it, for people. +func ReadExportState( + implementation configv1alpha1.MCSImplementation, + serviceExport *unstructured.Unstructured, +) (ExportState, string) { + var status struct { + Conditions []metav1.Condition `json:"conditions"` + } + if fields, ok := serviceExport.Object["status"].(map[string]any); ok { + // A status that cannot be read counts as no report. + _ = runtime.DefaultUnstructuredConverter.FromUnstructured(fields, &status) + } + + if valid := meta.FindStatusCondition(status.Conditions, "Valid"); valid != nil && + valid.Status == metav1.ConditionFalse { + return ExportRejected, describeCondition(valid) + } + // Pending is a reason for Ready to be False or Unknown while the export + // is in progress. + if ready := meta.FindStatusCondition(status.Conditions, "Ready"); ready != nil { + switch { + case ready.Status == metav1.ConditionFalse && ready.Reason != "Pending": + return ExportRejected, describeCondition(ready) + case ready.Status != metav1.ConditionTrue: + return ExportPending, describeCondition(ready) + } + } + if conflict := meta.FindStatusCondition(status.Conditions, "Conflict"); conflict != nil && + conflict.Status == metav1.ConditionTrue { + return ExportConflicting, describeCondition(conflict) + } + if len(status.Conditions) == 0 && implementation != configv1alpha1.MCSImplementationKarmada { + return ExportPending, "no conditions yet" + } + return ExportAccepted, "" +} + +// describeCondition returns condition as ` is (): `, +// leaving out a missing reason or message. +func describeCondition(condition *metav1.Condition) string { + description := fmt.Sprintf("%s is %s", condition.Type, condition.Status) + if condition.Reason != "" { + description += fmt.Sprintf(" (%s)", condition.Reason) + } + if condition.Message != "" { + description += ": " + condition.Message + } + return description +} + +// ReadClusterID returns the cluster's ID in its ClusterSet, from the About API's +// ClusterProperty named ClusterIDProperty, or "" when the cluster has no such +// property or the reader may not read it, as in a namespace-restricted install. +// reader should bypass the cache: the property is cluster-scoped, and reading it +// is optional. +func ReadClusterID(ctx context.Context, reader client.Reader) (string, error) { + property := &unstructured.Unstructured{} + property.SetGroupVersionKind(ClusterPropertyGVK) + err := reader.Get(ctx, client.ObjectKey{Name: ClusterIDProperty}, property) + if apierrors.IsNotFound(err) || apierrors.IsForbidden(err) || meta.IsNoMatchError(err) { + return "", nil + } + if err != nil { + return "", fmt.Errorf("failed to read ClusterProperty %s: %w", ClusterIDProperty, err) + } + value, _, _ := unstructured.NestedString(property.Object, "spec", "value") + return value, nil +} diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go index 7bd3ae78d48e..5ec738280ae5 100644 --- a/deploy/operator/internal/multicluster/serviceexport_test.go +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -6,34 +6,28 @@ package multicluster import ( + "context" + "errors" + "maps" "slices" + "strings" "testing" + "time" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" ) -func TestIsPool(t *testing.T) { - for _, tc := range []struct { - name string - annotations map[string]string - want bool - }{ - {name: "no annotations"}, - {name: "marked", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "true"}, want: true}, - {name: "marked in another case", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "True"}, want: true}, - {name: "marked false", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "false"}}, - {name: "not a boolean", annotations: map[string]string{commonconsts.KubeAnnotationGlobalRouterPool: "yes"}}, - } { - t.Run(tc.name, func(t *testing.T) { - t.Log("Read the Global Router pool mark") - if got := IsPool(tc.annotations); got != tc.want { - t.Errorf("IsPool(%v) = %v, want %v", tc.annotations, got, tc.want) - } - }) - } -} - func TestMaxServiceNameLength(t *testing.T) { for _, tc := range []struct { implementation configv1alpha1.MCSImplementation @@ -52,17 +46,60 @@ func TestMaxServiceNameLength(t *testing.T) { } func TestServiceNames(t *testing.T) { + component := func(name string, componentType v1beta1.ComponentType) v1beta1.DynamoComponentDeploymentSharedSpec { + return v1beta1.DynamoComponentDeploymentSharedSpec{ComponentName: name, ComponentType: componentType} + } for _, tc := range []struct { - dgdName string - want []string + name string + dgdName string + components []v1beta1.DynamoComponentDeploymentSharedSpec + wantFrontends []string }{ - {dgdName: "qwen", want: []string{"qwen-pool-relay", "qwen-frontend"}}, - {dgdName: "qwen3-0.6b", want: []string{"qwen3-0-6b-pool-relay", "qwen3-0-6b-frontend"}}, + { + name: "a frontend named Frontend gets the Service the Global Discovery Service expects by default", + dgdName: "qwen", + components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + component("Frontend", v1beta1.ComponentTypeFrontend), + component("VllmDecodeWorker", v1beta1.ComponentTypeDecode), + }, + wantFrontends: []string{"qwen-frontend"}, + }, + { + name: "every frontend keeps its component's name, lowercased, with dots replaced", + dgdName: "qwen3-0.6b", + components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + component("Web.v2", v1beta1.ComponentTypeFrontend), + component("Frontend", v1beta1.ComponentTypeFrontend), + }, + wantFrontends: []string{"qwen3-0-6b-web-v2", "qwen3-0-6b-frontend"}, + }, + { + name: "frontends whose Services share a name are listed once", + dgdName: "qwen", + components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + component("Frontend", v1beta1.ComponentTypeFrontend), + component("frontend", v1beta1.ComponentTypeFrontend), + }, + wantFrontends: []string{"qwen-frontend"}, + }, + { + name: "a DGD without a frontend has only its PoolRelay Service", + dgdName: "qwen", + components: []v1beta1.DynamoComponentDeploymentSharedSpec{component("Worker", v1beta1.ComponentTypeWorker)}, + }, } { - t.Run(tc.dgdName, func(t *testing.T) { - t.Log("Name the pool's Services as the operator names its own") - if got := ServiceNames(tc.dgdName); !slices.Equal(got, tc.want) { - t.Errorf("ServiceNames(%q) = %v, want %v", tc.dgdName, got, tc.want) + t.Run(tc.name, func(t *testing.T) { + dgd := &v1beta1.DynamoGraphDeployment{Name: tc.dgdName, Spec: v1beta1.DynamoGraphDeploymentSpec{Components: tc.components}} + + t.Log("Name the frontend Services as the operator names its own") + if got := FrontendServiceNames(dgd); !slices.Equal(got, tc.wantFrontends) { + t.Errorf("FrontendServiceNames = %v, want %v", got, tc.wantFrontends) + } + + t.Log("The pool's Services are its PoolRelay Service, then its frontend Services") + want := append([]string{tc.dgdName + "-pool-relay"}, tc.wantFrontends...) + if got := ServiceNames(dgd); !slices.Equal(got, want) { + t.Errorf("ServiceNames = %v, want %v", got, want) } }) } @@ -81,3 +118,240 @@ func TestRenderServiceExport(t *testing.T) { t.Errorf("DGD label = %q, want qwen", got) } } + +// exportWithConditions returns a ServiceExport whose status holds conditions, +// each given as type, status, reason and message. +func exportWithConditions(conditions ...[4]string) *unstructured.Unstructured { + export := RenderServiceExport("dynamo", "qwen-pool-relay", "qwen") + items := make([]any, 0, len(conditions)) + for _, condition := range conditions { + items = append(items, map[string]any{ + "type": condition[0], "status": condition[1], "reason": condition[2], "message": condition[3], + }) + } + if len(items) > 0 { + export.Object["status"] = map[string]any{"conditions": items} + } + return export +} + +func TestReadExportState(t *testing.T) { + const ( + standard = configv1alpha1.MCSImplementationStandard + karmada = configv1alpha1.MCSImplementationKarmada + ) + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + export *unstructured.Unstructured + want ExportState + wantDetail string + }{ + {name: "Karmada writes no status", implementation: karmada, export: exportWithConditions(), want: ExportAccepted}, + { + name: "a standard implementation has not reported yet", + implementation: standard, + export: exportWithConditions(), + want: ExportPending, + wantDetail: "no conditions yet", + }, + { + name: "valid, as an implementation without Ready reports it", + implementation: standard, + export: exportWithConditions([4]string{"Valid", "True", "Valid", ""}), + want: ExportAccepted, + }, + { + name: "valid and imported", + implementation: standard, + export: exportWithConditions( + [4]string{"Valid", "True", "Valid", ""}, + [4]string{"Ready", "True", "Exported", ""}, + [4]string{"Conflict", "False", "NoConflicts", ""}), + want: ExportAccepted, + }, + { + name: "a missing Service is invalid", + implementation: standard, + export: exportWithConditions([4]string{"Valid", "False", "NoService", "Service qwen-pool-relay not found"}), + want: ExportRejected, + wantDetail: "Valid is False (NoService): Service qwen-pool-relay not found", + }, + { + name: "an export the registry refused", + implementation: standard, + export: exportWithConditions([4]string{"Ready", "False", "Failed", ""}), + want: ExportRejected, + wantDetail: "Ready is False (Failed)", + }, + { + name: "an import that has not finished, reported as Unknown", + implementation: standard, + export: exportWithConditions([4]string{"Valid", "True", "", ""}, [4]string{"Ready", "Unknown", "Pending", ""}), + want: ExportPending, + wantDetail: "Ready is Unknown (Pending)", + }, + { + name: "an import that has not finished, reported as False", + implementation: standard, + export: exportWithConditions([4]string{"Valid", "True", "", ""}, [4]string{"Ready", "False", "Pending", ""}), + want: ExportPending, + wantDetail: "Ready is False (Pending)", + }, + { + name: "a conflict that the implementation resolved", + implementation: standard, + export: exportWithConditions( + [4]string{"Valid", "True", "", ""}, + [4]string{"Ready", "True", "Exported", ""}, + [4]string{"Conflict", "True", "PortConflict", "ports differ"}), + want: ExportConflicting, + wantDetail: "Conflict is True (PortConflict): ports differ", + }, + { + name: "a conflict while the import is not finished", + implementation: standard, + export: exportWithConditions( + [4]string{"Ready", "Unknown", "Pending", ""}, + [4]string{"Conflict", "True", "PortConflict", ""}), + want: ExportPending, + wantDetail: "Ready is Unknown (Pending)", + }, + { + name: "a status that cannot be read counts as no report", + implementation: standard, + export: func() *unstructured.Unstructured { + export := exportWithConditions() + export.Object["status"] = map[string]any{"conditions": "not a list"} + return export + }(), + want: ExportPending, + wantDetail: "no conditions yet", + }, + } { + t.Run(tc.name, func(t *testing.T) { + t.Log("Read what the MCS implementation reports on the export") + got, detail := ReadExportState(tc.implementation, tc.export) + if got != tc.want || detail != tc.wantDetail { + t.Errorf("ReadExportState = %v, %q, want %v, %q", got, detail, tc.want, tc.wantDetail) + } + }) + } +} + +func TestCheckServiceName(t *testing.T) { + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + service string + want NameProblem + }{ + {name: "a Service name", implementation: configv1alpha1.MCSImplementationKarmada, service: "qwen-pool-relay", want: NameExportable}, + {name: "a dot", implementation: configv1alpha1.MCSImplementationStandard, service: "qwen3-0.6b-pool-relay", want: NameInvalid}, + {name: "a leading digit", implementation: configv1alpha1.MCSImplementationStandard, service: "3b-frontend", want: NameInvalid}, + {name: "longer than Karmada imports", implementation: configv1alpha1.MCSImplementationKarmada, service: strings.Repeat("a", 56), want: NameTooLong}, + {name: "as long as a Service name gets", implementation: configv1alpha1.MCSImplementationStandard, service: strings.Repeat("a", 63), want: NameExportable}, + {name: "longer than any Service name", implementation: configv1alpha1.MCSImplementationStandard, service: strings.Repeat("a", 64), want: NameInvalid}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Log("Check whether the implementation can export a Service of the name") + got, detail := CheckServiceName(tc.implementation, tc.service) + if got != tc.want || (got == NameExportable) != (detail == "") { + t.Errorf("CheckServiceName(%q) = %v, %q, want %v", tc.service, got, detail, tc.want) + } + }) + } +} + +func TestListExports(t *testing.T) { + t.Log("Store an export controlled by a DGD that is being deleted, and one a user created that Karmada reports nothing on") + owned := exportWithConditions([4]string{"Valid", "False", "NoService", ""}) + owned.SetUID("export-1") + owned.SetOwnerReferences([]metav1.OwnerReference{ + {APIVersion: "nvidia.com/v1beta1", Kind: "DynamoGraphDeployment", Name: "qwen", UID: "dgd-1", Controller: new(true)}, + }) + owned.SetFinalizers([]string{"example.com/hold"}) + owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) + foreign := RenderServiceExport("dynamo", "qwen-frontend", "qwen") + foreign.SetUID("export-2") + elsewhere := RenderServiceExport("other", "qwen-frontend", "qwen") + mapper := meta.NewDefaultRESTMapper(nil) + mapper.Add(ServiceExportGVK, meta.RESTScopeNamespace) + reader := fake.NewClientBuilder(). + WithScheme(runtime.NewScheme()). + WithRESTMapper(mapper). + WithObjects(owned, foreign, elsewhere). + Build() + + t.Log("List the exports in namespace dynamo, as Karmada sees them") + got, err := ListExports(context.Background(), reader, "dynamo", configv1alpha1.MCSImplementationKarmada) + if err != nil { + t.Fatalf("ListExports: %v", err) + } + want := map[string]Export{ + "qwen-pool-relay": { + Name: "qwen-pool-relay", UID: "export-1", ControllerUID: "dgd-1", Deleting: true, + State: ExportRejected, Detail: "Valid is False (NoService)", + }, + "qwen-frontend": {Name: "qwen-frontend", UID: "export-2", State: ExportAccepted}, + } + if !maps.Equal(got, want) { + t.Errorf("ListExports = %+v, want %+v", got, want) + } +} + +func TestReadClusterID(t *testing.T) { + property := &unstructured.Unstructured{Object: map[string]any{"spec": map[string]any{"value": "west"}}} + property.SetGroupVersionKind(ClusterPropertyGVK) + property.SetName(ClusterIDProperty) + otherProperty := property.DeepCopy() + otherProperty.SetName("clusterset.k8s.io") + failure := errors.New("connection refused") + + for _, tc := range []struct { + name string + aboutAPI bool + objects []client.Object + getError error + want string + wantError bool + }{ + {name: "the cluster's ID in its ClusterSet", aboutAPI: true, objects: []client.Object{property, otherProperty}, want: "west"}, + {name: "no cluster ID property", aboutAPI: true, objects: []client.Object{otherProperty}}, + {name: "no About API"}, + { + name: "no permission, as in a namespace-restricted install", aboutAPI: true, + getError: apierrors.NewForbidden(schema.GroupResource{Group: "about.k8s.io", Resource: "clusterproperties"}, ClusterIDProperty, failure), + }, + {name: "another failure", aboutAPI: true, getError: failure, wantError: true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Log("Serve the About API, or not, and fail reads as the case needs") + mapper := meta.NewDefaultRESTMapper(nil) + if tc.aboutAPI { + mapper.Add(ClusterPropertyGVK, meta.RESTScopeRoot) + } + funcs := interceptor.Funcs{} + if tc.getError != nil { + funcs.Get = func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { + return tc.getError + } + } + reader := fake.NewClientBuilder(). + WithScheme(runtime.NewScheme()). + WithRESTMapper(mapper). + WithObjects(tc.objects...). + WithInterceptorFuncs(funcs). + Build() + + t.Log("Read the cluster's ID in its ClusterSet") + got, err := ReadClusterID(context.Background(), reader) + if (err != nil) != tc.wantError || got != tc.want { + t.Errorf("ReadClusterID = %q, %v, want %q with error %v", got, err, tc.want, tc.wantError) + } + if tc.wantError && !strings.Contains(err.Error(), ClusterIDProperty) { + t.Errorf("error %q does not name the property", err) + } + }) + } +} diff --git a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md index f79ef37f4434..8a8b2174d29c 100644 --- a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md +++ b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md @@ -382,6 +382,17 @@ and allow long-lived server streams. See the [gRPC contract](https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/pool_relay/docs/grpc-contract.md) for message sizes, reconnect behavior, and error reasons. +### Export to Other Clusters with MCS + +When the operator is configured with a Multi-Cluster Services implementation +(`dynamo-operator.dynamo.multiCluster.mcsImplementation`), a `DynamoPoolRelay` also makes its DGD a +Global Router pool. The operator exports `-pool-relay` and the DGD's frontend Services to the +other clusters of the ClusterSet. With a standard MCS implementation, set `clusterId` to the +cluster's ID in its ClusterSet, which the About API's `ClusterProperty` `cluster.clusterset.k8s.io` +holds. The DGD's `GlobalRouterPoolExported` condition reports a mismatch when the operator can read +that property. See +[Global Router pool export](../../reference/kubernetes-api/dynamo-graph-deployment.mdx#global-router-pool-export). + ## Optional mTLS Sidecar Mutual TLS (mTLS) is optional and implemented outside Relay. Relay has no built-in TLS configuration, diff --git a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md index 69f5c67f0a2e..4108eb17b321 100644 --- a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md +++ b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md @@ -4346,8 +4346,7 @@ _Appears in:_ MultiClusterConfiguration configures how the operator exports Global Router pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global Discovery Service's Kubernetes stores. The operator exports a pool only when -MCSImplementation is set and its DGD has the nvidia.com/global-router-pool -annotation. +MCSImplementation is set and a DynamoPoolRelay with its DGD's name exists. diff --git a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx index 90f7cba6e53a..1c2cc111f87f 100644 --- a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx +++ b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx @@ -145,9 +145,23 @@ Both fields retain their native hub/spoke conversion mapping. Converting between ## Global Router pool export -Set `metadata.annotations["nvidia.com/global-router-pool"]: "true"` to mark the DGD as a Global Router pool. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the Services `-pool-relay` and `-frontend`, with any dots in the DGD name replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. A [`DynamoPoolRelay`](../../kubernetes/kv-aware-routing/pool-relay.md) with the DGD's name creates `-pool-relay`, and the operator creates `-frontend` for a frontend component named `Frontend`. - -The DGD owns these exports: removing the annotation withdraws them, and deleting the DGD deletes them. A `ServiceExport` of the same name that the DGD does not own is left alone. With `karmada`, a Service name longer than 55 characters is not exported, because Karmada cannot import it. Without a configured implementation, the annotation has no effect. +A DGD is a Global Router pool while a [`DynamoPoolRelay`](../../kubernetes/kv-aware-routing/pool-relay.md) with its name exists in its namespace. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the PoolRelay Service, `-pool-relay`, and for the Service of each frontend component, `-` in lowercase with dots replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. It finds the frontends by a Service name suffix, `-frontend` by default, which matches a frontend component named `Frontend`. The operator does not know that suffix, so a frontend component with another name is exported but not found until the suffix matches it. + +The DGD owns these exports: deleting the `DynamoPoolRelay` withdraws them, renaming or removing a frontend component withdraws that component's export, and deleting the DGD deletes them. A `ServiceExport` of the same name that the DGD does not own is left alone. Without a configured implementation, nothing is exported. + +The `GlobalRouterPoolExported` condition reports on a pool's exports. It never fails the DGD or holds back its workloads. A new problem is also recorded as a Warning event on the DGD. + +| Reason | Status | Meaning | +| --- | --- | --- | +| `Exported` | True | Every Service is exported, and the MCS implementation reports no problem. Karmada reports nothing on its exports. | +| `ExportConflict` | True | Every Service is exported, but the MCS implementation reports `Conflict` as True: a property of an export differs from another cluster's export of the same Service, and the implementation resolved it. The message quotes that condition. | +| `ExportPending` | Unknown | An export was just created or is waiting to be replaced, a standard implementation has not reported on it yet, or it reports `Ready` as Unknown, or as False with the reason `Pending`. | +| `ExportRejected` | False | The MCS implementation reports `Valid` as False, or `Ready` as False for a reason other than `Pending`, on an export. The message quotes that condition. | +| `InvalidServiceName` | False | No Service can have a name that the pool needs, so that Service is not exported. For example, a DGD whose name has a dot has no PoolRelay Service. | +| `ServiceNameTooLong` | False | With `karmada`, a Service name is longer than 55 characters, which Karmada cannot import, so that Service is not exported. | +| `NoFrontend` | False | The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests. | +| `ClusterIDMismatch` | False | With `mcs`, the `DynamoPoolRelay`'s `clusterId` differs from the cluster's ID in its ClusterSet, which the About API's `ClusterProperty` `cluster.clusterset.k8s.io` holds. The Global Discovery Service lists the pool only when they match. The operator reads the ID at most once a minute, and skips the check when the cluster has no such property or the operator may not read it, as in a namespace-restricted install. | +| `ExportFailed` | False | Listing, creating or deleting an export failed. The operator retries within 30 seconds, or, when the DGD's own reconcile fails, the next time it reconciles the DGD. | ## Spec reference diff --git a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx index 36acb9411e51..88625e82a253 100644 --- a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx +++ b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx @@ -5947,8 +5947,7 @@ Secure enables secure serving for the metrics endpoint. nil = default to true (s MultiClusterConfiguration configures how the operator exports Global Router pools through the Kubernetes Multi-Cluster Services (MCS) API, for the Global Discovery Service's Kubernetes stores. The operator exports a pool only when -MCSImplementation is set and its DGD has the nvidia.com/global-router-pool -annotation. +MCSImplementation is set and a DynamoPoolRelay with its DGD's name exists. **Kind:** `type` From b53054efd205e2cf16bd646d3c1b93561d55fedc Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Fri, 30 Oct 2026 12:08:23 -0700 Subject: [PATCH 6/9] refactor(operator): resolve the ServiceExport version once and pass it on The multiClusterExport gate now returns the multicluster.x-k8s.io version at which the cluster serves ServiceExports, and Gates carries it as ServiceExportVersion. The export reconciler, the ServiceExport watch, and every list, create and delete use that version, through multicluster.ServiceExportGroupKind and version arguments. This follows the shape of #17606, which lets a cluster that serves only v1beta1 export. The gate here still probes only v1alpha1, so behavior does not change, and #17606 can add v1beta1 on top of this with its probe, its tests and its docs. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- .../dgd_service_export_reconciler.go | 17 +-- .../dgd_service_export_reconciler_test.go | 132 +++++++++++------- .../dynamographdeployment_controller.go | 2 +- deploy/operator/internal/features/gates.go | 13 +- .../internal/features/multicluster.go | 20 ++- .../internal/features/multicluster_test.go | 12 +- .../internal/multicluster/serviceexport.go | 42 +++--- .../multicluster/serviceexport_test.go | 18 +-- 8 files changed, 159 insertions(+), 97 deletions(-) diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler.go b/deploy/operator/internal/controller/dgd_service_export_reconciler.go index 20dd2f337df7..336ec7fa4220 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler.go @@ -80,11 +80,12 @@ const clusterSetIDTTL = time.Minute // that is a Global Router pool through the configured Multi-Cluster Services // implementation, and withdraws the exports when it stops being one. A DGD is a // pool while a DynamoPoolRelay of the same name exists. It creates and deletes -// only ServiceExports that the DGD controls. +// only ServiceExports that the DGD controls, at the multicluster.x-k8s.io version +// that the gates resolved. type dgdServiceExportReconciler struct { dgdResourceSyncer config *configv1alpha1.OperatorConfiguration - gate features.Gate + gates features.Gates // exports reads ServiceExports, which are unstructured: the manager's client // would read them from the API server, so this is the manager's cache, filled // by the controller's ServiceExport watch. @@ -97,12 +98,12 @@ type dgdServiceExportReconciler struct { func newDGDServiceExportReconciler( syncer dgdResourceSyncer, config *configv1alpha1.OperatorConfiguration, - gate features.Gate, + gates features.Gates, exports client.Reader, clusterSetID func(context.Context) string, ) *dgdServiceExportReconciler { return &dgdServiceExportReconciler{ - dgdResourceSyncer: syncer, config: config, gate: gate, exports: exports, clusterSetID: clusterSetID, + dgdResourceSyncer: syncer, config: config, gates: gates, exports: exports, clusterSetID: clusterSetID, } } @@ -116,7 +117,7 @@ func (r *dgdServiceExportReconciler) Reconcile( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, ) (*metav1.Condition, error) { - if !r.gate.Enabled(features.MultiClusterExport) { + if !r.gates.Enabled(features.MultiClusterExport) { return nil, nil } relay, err := r.poolRelay(ctx, dgd) @@ -146,7 +147,7 @@ func (r *dgdServiceExportReconciler) reconcileExports( relay *nvidiacomv1alpha1.DynamoPoolRelay, ) (*metav1.Condition, error) { implementation := r.config.MultiCluster.MCSImplementation - existing, err := multicluster.ListExports(ctx, r.exports, dgd.Namespace, implementation) + existing, err := multicluster.ListExports(ctx, r.exports, r.gates.ServiceExportVersion, dgd.Namespace, implementation) if err != nil { return nil, err } @@ -213,7 +214,7 @@ func (r *dgdServiceExportReconciler) export( name string, ) error { // Owned by the DGD, the export is deleted with it. - serviceExport := multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name) + serviceExport := multicluster.RenderServiceExport(r.gates.ServiceExportVersion, dgd.Namespace, name, dgd.Name) if err := ctrl.SetControllerReference(dgd, serviceExport, r.Scheme()); err != nil { return fmt.Errorf("failed to set the owner of ServiceExport %s: %w", name, err) } @@ -242,7 +243,7 @@ func (r *dgdServiceExportReconciler) withdraw( } // The UID precondition deletes exactly the export just read. - serviceExport := multicluster.NewServiceExport() + serviceExport := multicluster.NewServiceExport(r.gates.ServiceExportVersion) serviceExport.SetNamespace(dgd.Namespace) serviceExport.SetName(export.Name) if err := r.Delete(ctx, serviceExport, client.Preconditions{UID: &export.UID}); err != nil { diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go index a4fb31ccedfc..ab20df79a8f9 100644 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go +++ b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go @@ -50,11 +50,16 @@ import ( "sigs.k8s.io/controller-runtime/pkg/reconcile" ) +// serviceExportTestVersion is the multicluster.x-k8s.io version that the tests' +// clusters serve. +const serviceExportTestVersion = "v1alpha1" + // newServiceExportTestClient returns a fake client that also serves the -// ServiceExport and ClusterProperty kinds, which the operator's scheme does not -// register, and the DGD's status subresource. +// ServiceExport kind, at serviceExportVersion only, and the ClusterProperty kind, +// which the operator's scheme does not register, and the DGD's status subresource. func newServiceExportTestClient( scheme *runtime.Scheme, + serviceExportVersion string, funcs interceptor.Funcs, objects ...client.Object, ) client.WithWatch { @@ -62,7 +67,7 @@ func newServiceExportTestClient( for gvk := range scheme.AllKnownTypes() { mapper.Add(gvk, meta.RESTScopeNamespace) } - mapper.Add(multicluster.ServiceExportGVK, meta.RESTScopeNamespace) + mapper.Add(multicluster.ServiceExportGroupKind.WithVersion(serviceExportVersion), meta.RESTScopeNamespace) mapper.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) return fake.NewClientBuilder(). WithScheme(scheme). @@ -73,11 +78,11 @@ func newServiceExportTestClient( Build() } -// listServiceExports returns the ServiceExports in namespace. -func listServiceExports(t *testing.T, c client.Client, namespace string) []unstructured.Unstructured { +// listServiceExports returns the ServiceExports in namespace, read at version. +func listServiceExports(t *testing.T, c client.Client, version, namespace string) []unstructured.Unstructured { t.Helper() list := &unstructured.UnstructuredList{} - list.SetGroupVersionKind(multicluster.ServiceExportGVK.GroupVersion().WithKind("ServiceExportList")) + list.SetGroupVersionKind(multicluster.ServiceExportGroupKind.WithVersion(version).GroupVersion().WithKind("ServiceExportList")) if err := c.List(context.Background(), list, client.InNamespace(namespace)); err != nil { t.Fatalf("list ServiceExports: %v", err) } @@ -108,16 +113,17 @@ func newPoolTestRelay(dgd *v1beta1.DynamoGraphDeployment) *v1alpha1.DynamoPoolRe } } -// ownedServiceExport returns the export of the Service named name that dgd -// controls. +// ownedServiceExport returns the export at version of the Service named name +// that dgd controls. func ownedServiceExport( t *testing.T, scheme *runtime.Scheme, + version string, dgd *v1beta1.DynamoGraphDeployment, name string, ) *unstructured.Unstructured { t.Helper() - export := multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name) + export := multicluster.RenderServiceExport(version, dgd.Namespace, name, dgd.Name) if err := controllerutil.SetControllerReference(dgd, export, scheme); err != nil { t.Fatalf("set the owner of ServiceExport %s: %v", name, err) } @@ -141,19 +147,24 @@ func withExportConditions(export *unstructured.Unstructured, conditions ...metav // imported. var imported = metav1.Condition{Type: "Ready", Status: metav1.ConditionTrue, Reason: "Exported"} -// newServiceExportTestReconciler returns the export reconciler with the gate -// that implementation resolves to, reading exports through c, in a cluster with -// clusterSetID as its ID in its ClusterSet. +// newServiceExportTestReconciler returns the export reconciler with the gates +// that implementation resolves to in a cluster serving ServiceExports at version, +// reading exports through c, in a cluster with clusterSetID as its ID in its +// ClusterSet. func newServiceExportTestReconciler( c client.Client, recorder events.EventRecorder, implementation configv1alpha1.MCSImplementation, + version string, clusterSetID string, ) *dgdServiceExportReconciler { config := &configv1alpha1.OperatorConfiguration{ MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: implementation}, } gates := features.Gates{MultiClusterExport: implementation != configv1alpha1.MCSImplementationNone} + if gates.MultiClusterExport { + gates.ServiceExportVersion = version + } return newDGDServiceExportReconciler(newDGDResourceSyncer(c, recorder), config, gates, c, func(context.Context) string { return clusterSetID }) } @@ -313,7 +324,7 @@ func TestDGDServiceExportReconciler(t *testing.T) { objects = append(objects, relay) } for _, name := range tc.ownedBefore { - owned := ownedServiceExport(t, scheme, dgd, name) + owned := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, name) if tc.terminating { owned.SetFinalizers([]string{"example.com/hold"}) owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) @@ -321,17 +332,17 @@ func TestDGDServiceExportReconciler(t *testing.T) { objects = append(objects, owned) } for _, name := range tc.foreign { - objects = append(objects, multicluster.RenderServiceExport(dgd.Namespace, name, dgd.Name)) + objects = append(objects, multicluster.RenderServiceExport(serviceExportTestVersion, dgd.Namespace, name, dgd.Name)) } another := newPoolTestDGD("another", "Frontend") for _, name := range tc.ownedByAnother { - objects = append(objects, ownedServiceExport(t, scheme, another, name)) + objects = append(objects, ownedServiceExport(t, scheme, serviceExportTestVersion, another, name)) } - fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, objects...) + fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, objects...) t.Log("Reconcile the exports, with the gate the configured implementation resolves to") recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, "") + reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, serviceExportTestVersion, "") _, err := reconciler.Reconcile(ctx, dgd) g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) @@ -350,7 +361,7 @@ func TestDGDServiceExportReconciler(t *testing.T) { t.Log("Check which exports the DGD controls, and that other exports are untouched") var owned, others []string - for i, export := range listServiceExports(t, fakeClient, dgd.Namespace) { + for i, export := range listServiceExports(t, fakeClient, serviceExportTestVersion, dgd.Namespace) { if !metav1.IsControlledBy(&export, dgd) { others = append(others, export.GetName()) continue @@ -364,6 +375,31 @@ func TestDGDServiceExportReconciler(t *testing.T) { } } +func TestDGDServiceExportReconcilerUsesTheResolvedVersion(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + const version = "v1beta1" + + t.Log("Serve ServiceExports at a version other than the tests' default, with an export to withdraw") + scheme := newDynamoGraphDeploymentControllerTestScheme(t) + dgd := newPoolTestDGD("qwen", "Web") + fakeClient := newServiceExportTestClient(scheme, version, interceptor.Funcs{}, + dgd, newPoolTestRelay(dgd), ownedServiceExport(t, scheme, version, dgd, "qwen-frontend")) + + t.Log("Reconcile with the gates resolved to that version: exports are listed, created and withdrawn at it") + recorder := events.NewFakeRecorder(10) + reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, version, "") + _, err := reconciler.Reconcile(ctx, dgd) + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(recorder.Events).To(gomega.HaveLen(3)) + var names []string + for _, export := range listServiceExports(t, fakeClient, version, dgd.Namespace) { + g.Expect(export.GetAPIVersion()).To(gomega.Equal("multicluster.x-k8s.io/" + version)) + names = append(names, export.GetName()) + } + g.Expect(names).To(gomega.ConsistOf("qwen-pool-relay", "qwen-web")) +} + func TestDGDServiceExportReconcilerReportsTheImport(t *testing.T) { for _, tc := range []struct { name string @@ -453,8 +489,8 @@ func TestDGDServiceExportReconcilerReportsTheImport(t *testing.T) { t.Log("Create a pool whose exports exist, with what the MCS implementation reported on them") scheme := newDynamoGraphDeploymentControllerTestScheme(t) dgd := newPoolTestDGD("qwen", "Frontend") - relayExport := ownedServiceExport(t, scheme, dgd, "qwen-pool-relay") - frontendExport := ownedServiceExport(t, scheme, dgd, "qwen-frontend") + relayExport := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-pool-relay") + frontendExport := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-frontend") if len(tc.relayConditions) > 0 { withExportConditions(relayExport, tc.relayConditions...) withExportConditions(frontendExport, imported) @@ -463,12 +499,12 @@ func TestDGDServiceExportReconcilerReportsTheImport(t *testing.T) { frontendExport.SetFinalizers([]string{"example.com/hold"}) frontendExport.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) } - fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, + fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, dgd, newPoolTestRelay(dgd), relayExport, frontendExport) t.Log("Reconcile and read the pool's condition") recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, "") + reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, serviceExportTestVersion, "") condition, err := reconciler.Reconcile(context.Background(), dgd) g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(recorder.Events).To(gomega.BeEmpty()) @@ -524,13 +560,13 @@ func TestDGDServiceExportReconcilerChecksTheClusterID(t *testing.T) { t.Log("Create a pool of cluster east whose exports are imported") scheme := newDynamoGraphDeploymentControllerTestScheme(t) dgd := newPoolTestDGD("qwen", "Frontend") - fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{}, + fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, dgd, newPoolTestRelay(dgd), - withExportConditions(ownedServiceExport(t, scheme, dgd, "qwen-pool-relay"), imported), - withExportConditions(ownedServiceExport(t, scheme, dgd, "qwen-frontend"), imported)) + withExportConditions(ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-pool-relay"), imported), + withExportConditions(ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-frontend"), imported)) t.Log("Reconcile in a cluster with the given ID in its ClusterSet, and read the pool's condition") - reconciler := newServiceExportTestReconciler(fakeClient, events.NewFakeRecorder(10), tc.implementation, tc.clusterSetID) + reconciler := newServiceExportTestReconciler(fakeClient, events.NewFakeRecorder(10), tc.implementation, serviceExportTestVersion, tc.clusterSetID) if tc.noClusterSetID { reconciler.clusterSetID = nil } @@ -556,7 +592,7 @@ func TestClusterSetIDs(t *testing.T) { property.SetName(multicluster.ClusterIDProperty) reads := 0 var failure error - reader := newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), interceptor.Funcs{ + reader := newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), serviceExportTestVersion, interceptor.Funcs{ Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { reads++ if failure != nil { @@ -588,15 +624,15 @@ func TestDGDServiceExportReconcilerToleratesAStaleCache(t *testing.T) { t.Log("The cache has not yet seen exports that the API server already has") scheme := newDynamoGraphDeploymentControllerTestScheme(t) dgd := newPoolTestDGD("qwen", "Frontend") - fakeClient := newServiceExportTestClient(scheme, interceptor.Funcs{ + fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { - return apierrors.NewAlreadyExists(multicluster.ServiceExportGVK.GroupVersion().WithResource("serviceexports").GroupResource(), obj.GetName()) + return apierrors.NewAlreadyExists(schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, obj.GetName()) }, }, dgd, newPoolTestRelay(dgd)) t.Log("Reconcile: AlreadyExists is no error, nothing is announced, and the exports are pending") recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, "") + reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, serviceExportTestVersion, "") condition, err := reconciler.Reconcile(ctx, dgd) g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(recorder.Events).To(gomega.BeEmpty()) @@ -614,7 +650,7 @@ func TestMapServiceExportToDGDRequests(t *testing.T) { elsewhere := newPoolTestDGD("llama", "Frontend") elsewhere.Namespace = "other" reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, interceptor.Funcs{}, llama, dotted, elsewhere), + Client: newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, llama, dotted, elsewhere), } for _, tc := range []struct { @@ -624,27 +660,27 @@ func TestMapServiceExportToDGDRequests(t *testing.T) { }{ { name: "a PoolRelay Service", - export: multicluster.RenderServiceExport("dynamo", "llama-pool-relay", "llama"), + export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama-pool-relay", "llama"), want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, }, { name: "a frontend Service under its component's name", - export: multicluster.RenderServiceExport("dynamo", "llama-web", "llama"), + export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama-web", "llama"), want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, }, { name: "the frontend Service of a DGD with a dot in its name", - export: multicluster.RenderServiceExport("dynamo", "qwen3-0-6b-frontend", "qwen3-0.6b"), + export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "qwen3-0-6b-frontend", "qwen3-0.6b"), want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(dotted)}}, }, { name: "an export the DGD controls for a frontend it no longer has", - export: ownedServiceExport(t, scheme, llama, "llama-old"), + export: ownedServiceExport(t, scheme, serviceExportTestVersion, llama, "llama-old"), want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, }, { name: "another Service", - export: multicluster.RenderServiceExport("dynamo", "llama", "llama"), + export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama", "llama"), want: []ctrl.Request{}, }, } { @@ -674,7 +710,7 @@ func TestPoolWatchPredicates(t *testing.T) { g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: relabeled})).To(gomega.BeFalse()) t.Log("A ServiceExport update counts when the MCS implementation reports or its deletion starts") - export := multicluster.RenderServiceExport("dynamo", "qwen-frontend", "qwen") + export := multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "qwen-frontend", "qwen") reported := withExportConditions(export.DeepCopy(), imported) relabeledExport := export.DeepCopy() relabeledExport.SetLabels(map[string]string{"team": "router"}) @@ -779,20 +815,22 @@ func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { refused := interceptor.Funcs{ Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { return apierrors.NewForbidden( - schema.GroupResource{Group: multicluster.ServiceExportGVK.Group, Resource: "serviceexports"}, + schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, "llama-pool-relay", errors.New("denied by policy")) }, } recorder := events.NewFakeRecorder(10) reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, refused, dgd, relay), + Client: newServiceExportTestClient(scheme, serviceExportTestVersion, refused, dgd, relay), Recorder: recorder, Config: &configv1alpha1.OperatorConfiguration{ MultiCluster: configv1alpha1.MultiClusterConfiguration{ MCSImplementation: configv1alpha1.MCSImplementationKarmada, }, }, - RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: features.Gates{MultiClusterExport: true}}, + RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: features.Gates{ + MultiClusterExport: true, ServiceExportVersion: serviceExportTestVersion, + }}, } t.Log("The failure asks for a retry, is recorded as a warning and in the condition, and returns no error that would fail the DGD") @@ -805,7 +843,7 @@ func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { t.Log("A failure to read the DynamoPoolRelay leaves the condition as it was, since the DGD may not be a pool") meta.SetStatusCondition(&dgd.Status.Conditions, *newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExported, "exported")) - reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{ + reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { if _, ok := obj.(*v1alpha1.DynamoPoolRelay); ok { return errors.New("cache not synced") @@ -822,18 +860,18 @@ func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { t.Log("A failed withdrawal from a DGD that is no longer a pool keeps the condition until the exports are gone") former := newPoolTestDGD("qwen", "Frontend") meta.SetStatusCondition(&former.Status.Conditions, *newPoolCondition(former, metav1.ConditionTrue, reasonPoolExported, "exported")) - reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{ + reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { return errors.New("denied by policy") }, - }, former, ownedServiceExport(t, scheme, former, "qwen-frontend")) + }, former, ownedServiceExport(t, scheme, serviceExportTestVersion, former, "qwen-frontend")) result = reconciler.reconcilePoolExports(ctx, former) g.Expect(result.retry).To(gomega.BeTrue()) g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExported)) g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) t.Log("Exports that succeed ask for no retry") - reconciler.Client = newServiceExportTestClient(scheme, interceptor.Funcs{}, dgd, relay) + reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, dgd, relay) result = reconciler.reconcilePoolExports(ctx, dgd) g.Expect(result.retry).To(gomega.BeFalse()) g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExportPending)) @@ -846,10 +884,10 @@ func newPoolTestDGDReconciler( gates features.Gates, objects ...client.Object, ) (*DynamoGraphDeploymentReconciler, *events.FakeRecorder) { - gates.MultiClusterExport = true + gates.MultiClusterExport, gates.ServiceExportVersion = true, serviceExportTestVersion recorder := events.NewFakeRecorder(10) return &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), interceptor.Funcs{}, objects...), + Client: newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), serviceExportTestVersion, interceptor.Funcs{}, objects...), Recorder: recorder, Config: &configv1alpha1.OperatorConfiguration{ MultiCluster: configv1alpha1.MultiClusterConfiguration{ @@ -916,7 +954,7 @@ func TestDynamoGraphDeploymentReconcilePersistsThePoolCondition(t *testing.T) { g.Expect(reconciler.Delete(ctx, relay)).To(gomega.Succeed()) stored, _ = reconcileAndRead() g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType)).To(gomega.BeNil()) - g.Expect(listServiceExports(t, reconciler.Client, dgd.Namespace)).To(gomega.BeEmpty()) + g.Expect(listServiceExports(t, reconciler.Client, serviceExportTestVersion, dgd.Namespace)).To(gomega.BeEmpty()) } func TestDynamoGraphDeploymentReconcileKeepsThePoolConditionOnEarlyFailures(t *testing.T) { diff --git a/deploy/operator/internal/controller/dynamographdeployment_controller.go b/deploy/operator/internal/controller/dynamographdeployment_controller.go index 97d9dd4075b6..1b2661e7c7cb 100644 --- a/deploy/operator/internal/controller/dynamographdeployment_controller.go +++ b/deploy/operator/internal/controller/dynamographdeployment_controller.go @@ -509,7 +509,7 @@ func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) err // DynamoPoolRelay makes the DGD of the same name a pool. if r.RuntimeConfig.Gate.Enabled(features.MultiClusterExport) { ctrlBuilder = ctrlBuilder.Watches( - multicluster.NewServiceExport(), + multicluster.NewServiceExport(r.RuntimeConfig.Gate.ServiceExportVersion), handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToDGDRequests), builder.WithPredicates(predicate.Funcs{ CreateFunc: func(ce event.CreateEvent) bool { return true }, diff --git a/deploy/operator/internal/features/gates.go b/deploy/operator/internal/features/gates.go index ac59b6f58762..21538ddc5717 100644 --- a/deploy/operator/internal/features/gates.go +++ b/deploy/operator/internal/features/gates.go @@ -157,7 +157,8 @@ type Gate interface { Enabled(Name) bool } -// Gates is the complete set of operator feature gates. +// Gates is the complete set of operator feature gates, with the API version +// that a gate resolved where its consumers depend on it. type Gates struct { Checkpoint bool `json:"checkpoint"` Grove bool `json:"grove"` @@ -170,6 +171,11 @@ type Gates struct { Istio bool `json:"istio"` MultiClusterExport bool `json:"multiClusterExport"` GPUDiscovery bool `json:"gpuDiscovery"` + + // ServiceExportVersion is not a gate. It is the multicluster.x-k8s.io + // version that the operator reads and writes ServiceExports in, set + // whenever MultiClusterExport is enabled. + ServiceExportVersion string `json:"-"` } // Defaults returns the default feature gates. @@ -287,11 +293,12 @@ func New( } // Export Global Router pools only through a configured MCS implementation whose - // ServiceExport API this cluster serves. + // ServiceExport API this cluster serves, at the version it serves. if config.MultiCluster.MCSImplementation != configv1alpha1.MCSImplementationNone { - if gates.MultiClusterExport, err = resolveMultiClusterExport(ctx, mgr.GetConfig()); err != nil { + if gates.ServiceExportVersion, err = resolveMultiClusterExport(ctx, mgr.GetConfig()); err != nil { return Gates{}, err } + gates.MultiClusterExport = true } logValues := make([]any, 0, 2*len(allNames)) diff --git a/deploy/operator/internal/features/multicluster.go b/deploy/operator/internal/features/multicluster.go index a48ca93a35de..27cdc4c3ed1d 100644 --- a/deploy/operator/internal/features/multicluster.go +++ b/deploy/operator/internal/features/multicluster.go @@ -5,6 +5,7 @@ package features import ( "context" + "errors" "k8s.io/client-go/rest" ) @@ -25,14 +26,19 @@ import ( const MultiClusterExport Name = "multiClusterExport" // resolveMultiClusterExport verifies API availability for a configured MCS -// implementation. A nil config retains the native discovery-unavailable error. -func resolveMultiClusterExport(ctx context.Context, config *rest.Config) (bool, error) { +// implementation and returns the multicluster.x-k8s.io version at which the +// cluster serves ServiceExports. A nil config retains the native +// discovery-unavailable error. +func resolveMultiClusterExport(ctx context.Context, config *rest.Config) (string, error) { // Every supported implementation reads the SIG Multicluster ServiceExport. - available, err := detectAPIAvailability(ctx, config, "multicluster.x-k8s.io", "v1alpha1", "serviceexports") + const version = "v1alpha1" + available, err := detectAPIAvailability(ctx, config, "multicluster.x-k8s.io", version, "serviceexports") if err != nil { - return false, err + return "", err } - - return resolve(new(true), available, - "multiCluster.mcsImplementation is set but the multicluster.x-k8s.io/v1alpha1 ServiceExport API was not detected in the cluster") + if !available { + return "", errors.New("multiCluster.mcsImplementation is set but the multicluster.x-k8s.io/v1alpha1 " + + "ServiceExport API was not detected in the cluster") + } + return version, nil } diff --git a/deploy/operator/internal/features/multicluster_test.go b/deploy/operator/internal/features/multicluster_test.go index f5cfc1467e26..d482dc5c7968 100644 --- a/deploy/operator/internal/features/multicluster_test.go +++ b/deploy/operator/internal/features/multicluster_test.go @@ -56,10 +56,14 @@ func TestResolveMultiClusterExport(t *testing.T) { })) defer server.Close() - t.Log("Require the ServiceExport resource and preserve discovery failures") - enabled, err := resolveMultiClusterExport(t.Context(), &rest.Config{Host: server.URL}) - if enabled != (tt.wantErr == "") || (err == nil) != (tt.wantErr == "") { - t.Fatalf("resolveMultiClusterExport() = %v, %v; want enabled=%v, error=%q", enabled, err, tt.wantErr == "", tt.wantErr) + t.Log("Require the ServiceExport resource, return its version, and preserve discovery failures") + version, err := resolveMultiClusterExport(t.Context(), &rest.Config{Host: server.URL}) + wantVersion := "" + if tt.wantErr == "" { + wantVersion = "v1alpha1" + } + if version != wantVersion || (err == nil) != (tt.wantErr == "") { + t.Fatalf("resolveMultiClusterExport() = %q, %v; want %q, error=%q", version, err, wantVersion, tt.wantErr) } if err != nil && err.Error() != tt.wantErr { t.Errorf("resolveMultiClusterExport() error = %q, want %q", err, tt.wantErr) diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 376d744cd01c..6282056a2dae 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -29,16 +29,16 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" ) -// ServiceExportGVK is the SIG Multicluster ServiceExport. Every supported +// ServiceExportGroupKind is the SIG Multicluster ServiceExport. Every supported // implementation reads it in the cluster that runs the Service, Karmada included: // its ServiceExport controller watches this kind in member clusters. Karmada // watches a member only while its control plane has propagated at least one // ServiceExport there, so keep one in place in each member. That setup is outside -// the operator. -var ServiceExportGVK = schema.GroupVersionKind{ - Group: "multicluster.x-k8s.io", - Version: "v1alpha1", - Kind: "ServiceExport", +// the operator. The operator reads and writes it at the multicluster.x-k8s.io +// version that the multiClusterExport feature gate resolved. +var ServiceExportGroupKind = schema.GroupKind{ + Group: "multicluster.x-k8s.io", + Kind: "ServiceExport", } // ClusterPropertyGVK is the About API's ClusterProperty (KEP-2149), which holds @@ -145,17 +145,21 @@ func UsesClusterSetID(implementation configv1alpha1.MCSImplementation) bool { return implementation == configv1alpha1.MCSImplementationStandard } -// NewServiceExport returns an empty ServiceExport, to watch exports or to name one. -func NewServiceExport() *unstructured.Unstructured { +// NewServiceExport returns an empty ServiceExport at version, to watch exports or +// to name one. version must be a multicluster.x-k8s.io version that the cluster +// serves. +func NewServiceExport(version string) *unstructured.Unstructured { serviceExport := &unstructured.Unstructured{} - serviceExport.SetGroupVersionKind(ServiceExportGVK) + serviceExport.SetGroupVersionKind(ServiceExportGroupKind.WithVersion(version)) return serviceExport } -// RenderServiceExport returns the ServiceExport for the Service named name in -// namespace, labeled with the DGD that exports it. It sets no owner reference. -func RenderServiceExport(namespace, name, dgdName string) *unstructured.Unstructured { - serviceExport := NewServiceExport() +// RenderServiceExport returns the ServiceExport at version for the Service named +// name in namespace, labeled with the DGD that exports it. It sets no owner +// reference. version must be a multicluster.x-k8s.io version that the cluster +// serves. +func RenderServiceExport(version, namespace, name, dgdName string) *unstructured.Unstructured { + serviceExport := NewServiceExport(version) serviceExport.SetNamespace(namespace) serviceExport.SetName(name) serviceExport.SetLabels(map[string]string{ @@ -197,17 +201,19 @@ type Export struct { Detail string } -// ListExports returns the ServiceExports in namespace by name, with what -// implementation reports on them. reader should serve unstructured objects from -// a cache: the manager's client reads them from the API server. +// ListExports returns the ServiceExports in namespace by name, read at version, +// with what implementation reports on them. reader should serve unstructured +// objects from a cache: the manager's client reads them from the API server. func ListExports( ctx context.Context, reader client.Reader, - namespace string, + version, namespace string, implementation configv1alpha1.MCSImplementation, ) (map[string]Export, error) { list := &unstructured.UnstructuredList{} - list.SetGroupVersionKind(ServiceExportGVK.GroupVersion().WithKind(ServiceExportGVK.Kind + "List")) + list.SetGroupVersionKind(schema.GroupVersionKind{ + Group: ServiceExportGroupKind.Group, Version: version, Kind: ServiceExportGroupKind.Kind + "List", + }) if err := reader.List(ctx, list, client.InNamespace(namespace)); err != nil { return nil, fmt.Errorf("failed to list ServiceExports: %w", err) } diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go index 5ec738280ae5..afc598e1ca60 100644 --- a/deploy/operator/internal/multicluster/serviceexport_test.go +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -106,10 +106,10 @@ func TestServiceNames(t *testing.T) { } func TestRenderServiceExport(t *testing.T) { - t.Log("Render the export of pool qwen's PoolRelay Service") - export := RenderServiceExport("dynamo", "qwen-pool-relay", "qwen") - if got := export.GroupVersionKind(); got != ServiceExportGVK { - t.Errorf("GroupVersionKind = %v, want %v", got, ServiceExportGVK) + t.Log("Render the export of pool qwen's PoolRelay Service at the version the gate resolved") + export := RenderServiceExport("v1alpha1", "dynamo", "qwen-pool-relay", "qwen") + if export.GetAPIVersion() != "multicluster.x-k8s.io/v1alpha1" || export.GetKind() != "ServiceExport" { + t.Errorf("export type = %s %s, want multicluster.x-k8s.io/v1alpha1 ServiceExport", export.GetAPIVersion(), export.GetKind()) } if export.GetNamespace() != "dynamo" || export.GetName() != "qwen-pool-relay" { t.Errorf("export = %s/%s, want dynamo/qwen-pool-relay", export.GetNamespace(), export.GetName()) @@ -122,7 +122,7 @@ func TestRenderServiceExport(t *testing.T) { // exportWithConditions returns a ServiceExport whose status holds conditions, // each given as type, status, reason and message. func exportWithConditions(conditions ...[4]string) *unstructured.Unstructured { - export := RenderServiceExport("dynamo", "qwen-pool-relay", "qwen") + export := RenderServiceExport("v1alpha1", "dynamo", "qwen-pool-relay", "qwen") items := make([]any, 0, len(conditions)) for _, condition := range conditions { items = append(items, map[string]any{ @@ -272,11 +272,11 @@ func TestListExports(t *testing.T) { }) owned.SetFinalizers([]string{"example.com/hold"}) owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) - foreign := RenderServiceExport("dynamo", "qwen-frontend", "qwen") + foreign := RenderServiceExport("v1alpha1", "dynamo", "qwen-frontend", "qwen") foreign.SetUID("export-2") - elsewhere := RenderServiceExport("other", "qwen-frontend", "qwen") + elsewhere := RenderServiceExport("v1alpha1", "other", "qwen-frontend", "qwen") mapper := meta.NewDefaultRESTMapper(nil) - mapper.Add(ServiceExportGVK, meta.RESTScopeNamespace) + mapper.Add(ServiceExportGroupKind.WithVersion("v1alpha1"), meta.RESTScopeNamespace) reader := fake.NewClientBuilder(). WithScheme(runtime.NewScheme()). WithRESTMapper(mapper). @@ -284,7 +284,7 @@ func TestListExports(t *testing.T) { Build() t.Log("List the exports in namespace dynamo, as Karmada sees them") - got, err := ListExports(context.Background(), reader, "dynamo", configv1alpha1.MCSImplementationKarmada) + got, err := ListExports(context.Background(), reader, "v1alpha1", "dynamo", configv1alpha1.MCSImplementationKarmada) if err != nil { t.Fatalf("ListExports: %v", err) } From b150caebd8f78d00ac3dd9f885c844196802c9de Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Mon, 2 Nov 2026 20:23:55 -0800 Subject: [PATCH 7/9] refactor(operator): maintain the pool's exports in the PoolRelay controller Move the pool's ServiceExports from the DGD controller into the DynamoPoolRelay controller, which already watches the Relay's DGD and publishes the Relay's Service. The exports now follow that decision: - The Relay controls each export it creates, so garbage collection deletes the exports with the Relay, and every reason the controller withdraws the Relay's Service, such as a missing or terminating DGD, unsupported discovery, no workers or an ownership conflict, withdraws the exports too. - The DGD controller no longer touches exports, and the DGD has no GlobalRouterPoolExported condition. The problems that only the operator can see are Warning events on the Relay: InvalidServiceName, ServiceNameTooLong, NoFrontend, ClusterIDMismatch and ExportFailed. The MCS implementation reports on each export in the export's own status. - A terminal error from the DGD's workloads no longer stops the exports from being retried: the Relay controller retries a failed export write with backoff, whatever the DGD's workloads report. - An export that the Relay controls is repaired with a server-side apply of only the fields that the controller writes. The apply carries the UID and resourceVersion of the export whose controller was checked, so it never writes to an export that changed hands in between. - An export is deleted with UID and resourceVersion preconditions, for the same reason. - The controller watches the About API's ClusterProperty that holds the cluster's ID in its ClusterSet, so a change of that ID is reported at once. Namespace-restricted installs read it through a cluster-wide reader role. The docs describe the exports on the PoolRelay page, and the DGD reference drops its pool export section. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- deploy/helm/charts/platform/README.md | 2 +- .../components/operator/files/role.yaml | 3 + .../operator/templates/manager-rbac.yaml | 39 + deploy/helm/charts/platform/values.yaml | 2 +- deploy/operator/api/config/v1alpha1/types.go | 4 +- .../dgd_service_export_reconciler.go | 437 ------- .../dgd_service_export_reconciler_test.go | 1007 --------------- .../controller/dgd_workload_program_test.go | 2 +- .../dynamographdeployment_controller.go | 155 +-- .../controller/poolrelay/controller.go | 65 +- .../internal/controller/poolrelay/exports.go | 515 ++++++++ .../poolrelay/exports_envtest_test.go | 446 +++++++ .../controller/poolrelay/exports_test.go | 1090 +++++++++++++++++ deploy/operator/internal/controller/setup.go | 3 - .../internal/multicluster/serviceexport.go | 130 +- .../multicluster/serviceexport_test.go | 147 +-- .../kubernetes/kv-aware-routing/pool-relay.md | 38 +- .../additional-resources/api-reference-k8s.md | 2 +- .../dynamo-graph-deployment.mdx | 20 - .../kubernetes-api/full-api-reference.mdx | 2 +- 20 files changed, 2243 insertions(+), 1866 deletions(-) delete mode 100644 deploy/operator/internal/controller/dgd_service_export_reconciler.go delete mode 100644 deploy/operator/internal/controller/dgd_service_export_reconciler_test.go create mode 100644 deploy/operator/internal/controller/poolrelay/exports.go create mode 100644 deploy/operator/internal/controller/poolrelay/exports_envtest_test.go create mode 100644 deploy/operator/internal/controller/poolrelay/exports_test.go diff --git a/deploy/helm/charts/platform/README.md b/deploy/helm/charts/platform/README.md index 5d181d7ef970..7fdd57863494 100644 --- a/deploy/helm/charts/platform/README.md +++ b/deploy/helm/charts/platform/README.md @@ -594,7 +594,7 @@ Kubernetes: `>=1.30.0-0` | dynamo-operator.dynamo.serviceMesh.istio.clientCertificate | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS certificate used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.privateKey | string | `""` | Path (in the istio-proxy sidecar's filesystem) to the client TLS private key used for mTLS. REQUIRED when tlsMode is "MUTUAL"; ignored for other modes. | | dynamo-operator.dynamo.serviceMesh.istio.caCertificates | string | `""` | Optional path (in the istio-proxy sidecar's filesystem) to CA certificates used to verify the EPP server certificate. Used only when tlsMode is "MUTUAL". | -| dynamo-operator.dynamo.multiCluster.mcsImplementation | string | `""` | MCS implementation that imports the exports: "mcs" for the SIG Multicluster API or "karmada". For each DGD with a DynamoPoolRelay of the same name, the operator then creates a ServiceExport for `-pool-relay` and for each frontend component's Service. Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this cluster. Empty exports nothing; clearing it leaves earlier exports in place until their DGDs are deleted. | +| dynamo-operator.dynamo.multiCluster.mcsImplementation | string | `""` | MCS implementation that imports the exports: "mcs" for the SIG Multicluster API or "karmada". For each DGD with a DynamoPoolRelay of the same name, the operator then creates a ServiceExport for `-pool-relay` and for each frontend component's Service. Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this cluster. Empty exports nothing; clearing it leaves earlier exports in place until their DynamoPoolRelays are deleted. | | dynamo-operator.dynamo.metrics.prometheusEndpoint | string | `""` | Endpoint that services can use to retrieve metrics. If set, dynamo operator will automatically inject the PROMETHEUS_ENDPOINT environment variable into services it manages. Users can override the value of the PROMETHEUS_ENDPOINT environment variable by modifying the corresponding deployment's environment variables | | dynamo-operator.dynamo.mpiRun.secretName | string | `"mpi-run-ssh-secret"` | Name of the secret containing the SSH key for MPI Run | | dynamo-operator.webhook.certificateSecret.name | string | `"webhook-server-cert"` | Name of the Kubernetes secret containing webhook TLS certificates. The secret must contain three keys: tls.crt (server certificate), tls.key (server private key), and ca.crt (Certificate Authority certificate). | diff --git a/deploy/helm/charts/platform/components/operator/files/role.yaml b/deploy/helm/charts/platform/components/operator/files/role.yaml index b522ff9693ec..ca0aa6f59527 100644 --- a/deploy/helm/charts/platform/components/operator/files/role.yaml +++ b/deploy/helm/charts/platform/components/operator/files/role.yaml @@ -82,6 +82,8 @@ rules: - clusterproperties verbs: - get + - list + - watch - apiGroups: - apiextensions.k8s.io resources: @@ -270,6 +272,7 @@ rules: - delete - get - list + - patch - watch - apiGroups: - networking.istio.io diff --git a/deploy/helm/charts/platform/components/operator/templates/manager-rbac.yaml b/deploy/helm/charts/platform/components/operator/templates/manager-rbac.yaml index 162adc60ef4b..f538241b0a43 100644 --- a/deploy/helm/charts/platform/components/operator/templates/manager-rbac.yaml +++ b/deploy/helm/charts/platform/components/operator/templates/manager-rbac.yaml @@ -314,3 +314,42 @@ subjects: - kind: ServiceAccount name: '{{ include "dynamo-operator.fullname" . }}-controller-manager' namespace: '{{ .Release.Namespace }}' +--- +# Helm keeps ClusterProperty access separate because a namespace-restricted Role cannot grant it. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "dynamo-operator.fullname" . }}-{{ .Release.Namespace }}-clusterproperty-reader + labels: + app.kubernetes.io/component: rbac + app.kubernetes.io/created-by: dynamo-operator + app.kubernetes.io/part-of: dynamo-operator + {{- include "dynamo-operator.labels" . | nindent 4 }} +rules: +- apiGroups: + - about.k8s.io + resources: + - clusterproperties + verbs: + - get + - list + - watch +--- +# Helm binds the separate cluster-scoped ClusterProperty grant to the controller ServiceAccount. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "dynamo-operator.fullname" . }}-{{ .Release.Namespace }}-clusterproperty-reader-binding + labels: + app.kubernetes.io/component: rbac + app.kubernetes.io/created-by: dynamo-operator + app.kubernetes.io/part-of: dynamo-operator + {{- include "dynamo-operator.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "dynamo-operator.fullname" . }}-{{ .Release.Namespace }}-clusterproperty-reader +subjects: +- kind: ServiceAccount + name: '{{ include "dynamo-operator.fullname" . }}-controller-manager' + namespace: '{{ .Release.Namespace }}' diff --git a/deploy/helm/charts/platform/values.yaml b/deploy/helm/charts/platform/values.yaml index 6df998f4c889..39aa81fe69a0 100644 --- a/deploy/helm/charts/platform/values.yaml +++ b/deploy/helm/charts/platform/values.yaml @@ -256,7 +256,7 @@ dynamo-operator: # `-pool-relay` and for each frontend component's Service. # Needs the multicluster.x-k8s.io/v1alpha1 ServiceExport API in this # cluster. Empty exports nothing; clearing it leaves earlier exports in - # place until their DGDs are deleted. + # place until their DynamoPoolRelays are deleted. mcsImplementation: "" # Metrics configuration diff --git a/deploy/operator/api/config/v1alpha1/types.go b/deploy/operator/api/config/v1alpha1/types.go index dcb74f94e45d..0539290cf8cf 100644 --- a/deploy/operator/api/config/v1alpha1/types.go +++ b/deploy/operator/api/config/v1alpha1/types.go @@ -407,8 +407,8 @@ type DiscoveryConfiguration struct { type MultiClusterConfiguration struct { // MCSImplementation is the MCS implementation that imports the exports. Empty, // the default, exports nothing: Global Router pools are then discovered without - // Kubernetes. Clearing it leaves earlier exports in place until their DGDs are - // deleted. + // Kubernetes. Clearing it leaves earlier exports in place until their + // DynamoPoolRelays are deleted. MCSImplementation MCSImplementation `json:"mcsImplementation,omitempty"` } diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler.go b/deploy/operator/internal/controller/dgd_service_export_reconciler.go deleted file mode 100644 index 336ec7fa4220..000000000000 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler.go +++ /dev/null @@ -1,437 +0,0 @@ -/* - * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package controller - -import ( - "context" - "fmt" - "maps" - "slices" - "strings" - "sync" - "time" - - configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" - nvidiacomv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" - nvidiacomv1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/equality" - apierrors "k8s.io/apimachinery/pkg/api/errors" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/event" - "sigs.k8s.io/controller-runtime/pkg/log" -) - -// globalRouterPoolExportedConditionType reports on a Global Router pool's DGD -// whether its Services are exported, as far as the operator and the MCS -// implementation can tell. -const globalRouterPoolExportedConditionType = "GlobalRouterPoolExported" - -// Reasons of the GlobalRouterPoolExported condition. -const ( - reasonPoolExported = "Exported" - reasonPoolExportConflict = "ExportConflict" - reasonPoolExportPending = "ExportPending" - reasonPoolExportFailed = "ExportFailed" - reasonPoolInvalidServiceName = "InvalidServiceName" - reasonPoolServiceNameTooLong = "ServiceNameTooLong" - reasonPoolNoFrontend = "NoFrontend" - reasonPoolClusterIDMismatch = "ClusterIDMismatch" - reasonPoolExportRejected = "ExportRejected" -) - -// poolProblemReasons are the reasons of the GlobalRouterPoolExported condition -// that are recorded as a Warning event when they are newly raised. A failed -// reconcile records its own event. -var poolProblemReasons = []string{ - reasonPoolExportConflict, - reasonPoolInvalidServiceName, - reasonPoolServiceNameTooLong, - reasonPoolNoFrontend, - reasonPoolClusterIDMismatch, - reasonPoolExportRejected, -} - -// clusterSetIDTTL is how long the operator reuses the cluster's ID in its -// ClusterSet before reading it again. -const clusterSetIDTTL = time.Minute - -// dgdServiceExportReconciler exports the PoolRelay and frontend Services of a DGD -// that is a Global Router pool through the configured Multi-Cluster Services -// implementation, and withdraws the exports when it stops being one. A DGD is a -// pool while a DynamoPoolRelay of the same name exists. It creates and deletes -// only ServiceExports that the DGD controls, at the multicluster.x-k8s.io version -// that the gates resolved. -type dgdServiceExportReconciler struct { - dgdResourceSyncer - config *configv1alpha1.OperatorConfiguration - gates features.Gates - // exports reads ServiceExports, which are unstructured: the manager's client - // would read them from the API server, so this is the manager's cache, filled - // by the controller's ServiceExport watch. - exports client.Reader - // clusterSetID returns the cluster's ID in its ClusterSet, or "" when it is - // unknown, which skips the check of the pool's cluster ID. - clusterSetID func(context.Context) string -} - -func newDGDServiceExportReconciler( - syncer dgdResourceSyncer, - config *configv1alpha1.OperatorConfiguration, - gates features.Gates, - exports client.Reader, - clusterSetID func(context.Context) string, -) *dgdServiceExportReconciler { - return &dgdServiceExportReconciler{ - dgdResourceSyncer: syncer, config: config, gates: gates, exports: exports, clusterSetID: clusterSetID, - } -} - -// Reconcile converges the DGD's ServiceExports on whether it is a pool, and -// returns the GlobalRouterPoolExported condition for the DGD, or nil when the DGD -// is not a pool or nothing is exported. A failure returns the error with an -// ExportFailed condition when the DGD is a pool, or with nil when that is -// unknown. Without a configured MCS implementation the operator exports nothing -// and leaves existing exports alone. -func (r *dgdServiceExportReconciler) Reconcile( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, -) (*metav1.Condition, error) { - if !r.gates.Enabled(features.MultiClusterExport) { - return nil, nil - } - relay, err := r.poolRelay(ctx, dgd) - if err != nil { - return nil, err - } - condition, err := r.reconcileExports(ctx, dgd, relay) - if err != nil && relay != nil { - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolExportFailed, "%v", err), err - } - return condition, err -} - -// unexportedService is a pool Service that the implementation cannot export. -type unexportedService struct { - name string - problem multicluster.NameProblem - detail string -} - -// reconcileExports exports the Services of the pool that relay makes the DGD, -// withdraws the DGD's other exports, and returns the pool's condition. A nil -// relay withdraws every export the DGD controls. -func (r *dgdServiceExportReconciler) reconcileExports( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - relay *nvidiacomv1alpha1.DynamoPoolRelay, -) (*metav1.Condition, error) { - implementation := r.config.MultiCluster.MCSImplementation - existing, err := multicluster.ListExports(ctx, r.exports, r.gates.ServiceExportVersion, dgd.Namespace, implementation) - if err != nil { - return nil, err - } - - // A pool exports each of its Services that the implementation can export. An - // existing export, the DGD's or a user's, needs nothing. - var exported []string - var unexported []unexportedService - if relay != nil { - for _, name := range multicluster.ServiceNames(dgd) { - if problem, detail := multicluster.CheckServiceName(implementation, name); problem != multicluster.NameExportable { - unexported = append(unexported, unexportedService{name: name, problem: problem, detail: detail}) - continue - } - if _, found := existing[name]; !found { - if err := r.export(ctx, dgd, name); err != nil { - return nil, err - } - } - exported = append(exported, name) - } - } - - // Withdraw the DGD's exports of Services it no longer exports: all of them once - // it is no longer a pool, and a renamed or removed frontend's. - for _, name := range slices.Sorted(maps.Keys(existing)) { - if !slices.Contains(exported, name) { - if err := r.withdraw(ctx, dgd, existing[name]); err != nil { - return nil, err - } - } - } - - if relay == nil { - return nil, nil - } - return r.condition(ctx, dgd, relay, exported, unexported, existing), nil -} - -// poolRelay returns the DynamoPoolRelay that makes the DGD a Global Router pool, -// or nil when there is none or it is being deleted. -func (r *dgdServiceExportReconciler) poolRelay( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, -) (*nvidiacomv1alpha1.DynamoPoolRelay, error) { - relay := &nvidiacomv1alpha1.DynamoPoolRelay{} - err := r.Get(ctx, client.ObjectKeyFromObject(dgd), relay) - if apierrors.IsNotFound(err) { - return nil, nil - } - if err != nil { - return nil, fmt.Errorf("failed to get DynamoPoolRelay %s: %w", dgd.Name, err) - } - if !relay.DeletionTimestamp.IsZero() { - return nil, nil - } - return relay, nil -} - -// export creates the DGD's ServiceExport for the Service named name. -func (r *dgdServiceExportReconciler) export( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - name string, -) error { - // Owned by the DGD, the export is deleted with it. - serviceExport := multicluster.RenderServiceExport(r.gates.ServiceExportVersion, dgd.Namespace, name, dgd.Name) - if err := ctrl.SetControllerReference(dgd, serviceExport, r.Scheme()); err != nil { - return fmt.Errorf("failed to set the owner of ServiceExport %s: %w", name, err) - } - if err := r.Create(ctx, serviceExport); err != nil { - // The cache has not seen an export created moments ago. Its watch event - // requeues the DGD. - if apierrors.IsAlreadyExists(err) { - return nil - } - return fmt.Errorf("failed to create ServiceExport %s: %w", name, err) - } - r.recorder.Eventf(dgd, nil, corev1.EventTypeNormal, "CreateServiceExport", "Create", - "Exported Service %s through MCS implementation %s", name, r.config.MultiCluster.MCSImplementation) - return nil -} - -// withdraw deletes export if the DGD controls it and it is not already being -// deleted. -func (r *dgdServiceExportReconciler) withdraw( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - export multicluster.Export, -) error { - if export.ControllerUID != dgd.UID || export.Deleting { - return nil - } - - // The UID precondition deletes exactly the export just read. - serviceExport := multicluster.NewServiceExport(r.gates.ServiceExportVersion) - serviceExport.SetNamespace(dgd.Namespace) - serviceExport.SetName(export.Name) - if err := r.Delete(ctx, serviceExport, client.Preconditions{UID: &export.UID}); err != nil { - if apierrors.IsNotFound(err) { - return nil - } - return fmt.Errorf("failed to delete ServiceExport %s: %w", export.Name, err) - } - r.recorder.Eventf(dgd, nil, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", - "Withdrew the export of Service %s", export.Name) - return nil -} - -// condition returns the pool's GlobalRouterPoolExported condition: the first -// problem that the operator or the MCS implementation finds, else whether every -// export is imported. existing holds the exports read before this reconcile -// created any. -func (r *dgdServiceExportReconciler) condition( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - relay *nvidiacomv1alpha1.DynamoPoolRelay, - exported []string, - unexported []unexportedService, - existing map[string]multicluster.Export, -) *metav1.Condition { - implementation := r.config.MultiCluster.MCSImplementation - if names, detail := unexportedNames(unexported, multicluster.NameInvalid); len(names) > 0 { - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolInvalidServiceName, - "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail) - } - if names, detail := unexportedNames(unexported, multicluster.NameTooLong); len(names) > 0 { - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolServiceNameTooLong, - "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", ")) - } - if len(multicluster.FrontendServiceNames(dgd)) == 0 { - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolNoFrontend, - "The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests") - } - - // A standard implementation labels the pool's imports with the cluster's ID - // in its ClusterSet, and the Global Discovery Service joins the pool's - // registration to them only under the same cluster ID. - if multicluster.UsesClusterSetID(implementation) && r.clusterSetID != nil { - if clusterID := r.clusterSetID(ctx); clusterID != "" && clusterID != relay.Spec.ClusterID { - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolClusterIDMismatch, - "DynamoPoolRelay %s has clusterId %q, but ClusterProperty %s gives this cluster the ID %q in its ClusterSet; "+ - "the Global Discovery Service lists the pool only when they match", - relay.Name, relay.Spec.ClusterID, multicluster.ClusterIDProperty, clusterID) - } - } - - var pending, conflicting []string - for _, name := range exported { - export, found := existing[name] - switch { - case !found || export.Deleting: - // Created by this reconcile, or waiting to be replaced. Its next watch - // event requeues the DGD. - pending = append(pending, name) - case export.State == multicluster.ExportRejected: - return newPoolCondition(dgd, metav1.ConditionFalse, reasonPoolExportRejected, - "%s reports on the export of Service %s: %s", implementation, name, export.Detail) - case export.State == multicluster.ExportPending: - pending = append(pending, name) - case export.State == multicluster.ExportConflicting: - conflicting = append(conflicting, fmt.Sprintf("%s (%s)", name, export.Detail)) - } - } - if len(pending) > 0 { - return newPoolCondition(dgd, metav1.ConditionUnknown, reasonPoolExportPending, - "Waiting for %s to import %s", implementation, strings.Join(pending, ", ")) - } - if len(conflicting) > 0 { - return newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExportConflict, - "Exported %s through %s, which resolves a conflict with another cluster's export of %s", - strings.Join(exported, ", "), implementation, strings.Join(conflicting, ", ")) - } - return newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExported, - "Exported %s through %s", strings.Join(exported, ", "), implementation) -} - -// unexportedNames returns the names of the services with problem, and the first -// one's detail. -func unexportedNames(services []unexportedService, problem multicluster.NameProblem) ([]string, string) { - var names []string - detail := "" - for _, service := range services { - if service.problem != problem { - continue - } - if len(names) == 0 { - detail = service.detail - } - names = append(names, service.name) - } - return names, detail -} - -// newPoolCondition returns the DGD's GlobalRouterPoolExported condition. -func newPoolCondition( - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - status metav1.ConditionStatus, - reason, format string, - args ...any, -) *metav1.Condition { - return &metav1.Condition{ - Type: globalRouterPoolExportedConditionType, - Status: status, - ObservedGeneration: dgd.Generation, - Reason: reason, - Message: fmt.Sprintf(format, args...), - } -} - -// clusterSetIDs keeps the cluster's ID in its ClusterSet, read through the API -// server at most once per clusterSetIDTTL: the cache does not hold it, and it -// rarely changes. -type clusterSetIDs struct { - mu sync.Mutex - id string - readAt time.Time -} - -// get returns the cluster's ID in its ClusterSet, or "" when it is unknown. A -// failed read is not kept, so the next call reads again. -func (c *clusterSetIDs) get(ctx context.Context, reader client.Reader, now time.Time) string { - c.mu.Lock() - defer c.mu.Unlock() - if !c.readAt.IsZero() && now.Sub(c.readAt) < clusterSetIDTTL { - return c.id - } - id, err := multicluster.ReadClusterID(ctx, reader) - if err != nil { - log.FromContext(ctx).V(1).Info("Failed to read the cluster's ID in its ClusterSet; skipping the check", "error", err) - return "" - } - c.id, c.readAt = id, now - return id -} - -// mapServiceExportToDGDRequests requeues each DGD in the export's namespace that -// controls the export or exports a Service of its name: the DGD withdraws an -// export it no longer needs, takes over when a user's export of the same Service -// is deleted, and follows what the MCS implementation reports. -func (r *DynamoGraphDeploymentReconciler) mapServiceExportToDGDRequests( - ctx context.Context, - obj client.Object, -) []ctrl.Request { - deployments := &nvidiacomv1beta1.DynamoGraphDeploymentList{} - if err := r.List(ctx, deployments, client.InNamespace(obj.GetNamespace())); err != nil { - log.FromContext(ctx).Error(err, "Failed to list DynamoGraphDeployments for ServiceExport event") - return nil - } - - owner := metav1.GetControllerOf(obj) - requests := make([]ctrl.Request, 0) - for i := range deployments.Items { - dgd := &deployments.Items[i] - controls := owner != nil && owner.UID == dgd.UID - if controls || slices.Contains(multicluster.ServiceNames(dgd), obj.GetName()) { - requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) - } - } - return requests -} - -// mapPoolRelayToDGDRequests requeues the DGD that a DynamoPoolRelay makes a -// Global Router pool: the one with the same name and namespace. -func mapPoolRelayToDGDRequests(_ context.Context, obj client.Object) []ctrl.Request { - return []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(obj)}} -} - -// serviceExportStateChanged passes the ServiceExport updates that can change a -// pool's condition: a new status from the MCS implementation, or the start of a -// deletion. -func serviceExportStateChanged(update event.UpdateEvent) bool { - oldExport, oldOK := update.ObjectOld.(*unstructured.Unstructured) - newExport, newOK := update.ObjectNew.(*unstructured.Unstructured) - if !oldOK || !newOK { - return false - } - return !equality.Semantic.DeepEqual(oldExport.Object["status"], newExport.Object["status"]) || - oldExport.GetDeletionTimestamp().IsZero() != newExport.GetDeletionTimestamp().IsZero() -} - -// poolRelayChanged passes the DynamoPoolRelay updates that can change its pool: -// a new spec, such as its cluster ID, or the start of a deletion. -func poolRelayChanged(update event.UpdateEvent) bool { - return update.ObjectOld.GetGeneration() != update.ObjectNew.GetGeneration() || - update.ObjectOld.GetDeletionTimestamp().IsZero() != update.ObjectNew.GetDeletionTimestamp().IsZero() -} diff --git a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go b/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go deleted file mode 100644 index ab20df79a8f9..000000000000 --- a/deploy/operator/internal/controller/dgd_service_export_reconciler_test.go +++ /dev/null @@ -1,1007 +0,0 @@ -/* - * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package controller - -import ( - "context" - "errors" - "strings" - "testing" - "time" - - configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" - "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" - "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" - commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" - "github.com/onsi/gomega" - corev1 "k8s.io/api/core/v1" - apierrors "k8s.io/apimachinery/pkg/api/errors" - "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/util/uuid" - "k8s.io/client-go/tools/events" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/client/fake" - "sigs.k8s.io/controller-runtime/pkg/client/interceptor" - "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" - "sigs.k8s.io/controller-runtime/pkg/event" - "sigs.k8s.io/controller-runtime/pkg/reconcile" -) - -// serviceExportTestVersion is the multicluster.x-k8s.io version that the tests' -// clusters serve. -const serviceExportTestVersion = "v1alpha1" - -// newServiceExportTestClient returns a fake client that also serves the -// ServiceExport kind, at serviceExportVersion only, and the ClusterProperty kind, -// which the operator's scheme does not register, and the DGD's status subresource. -func newServiceExportTestClient( - scheme *runtime.Scheme, - serviceExportVersion string, - funcs interceptor.Funcs, - objects ...client.Object, -) client.WithWatch { - mapper := meta.NewDefaultRESTMapper(nil) - for gvk := range scheme.AllKnownTypes() { - mapper.Add(gvk, meta.RESTScopeNamespace) - } - mapper.Add(multicluster.ServiceExportGroupKind.WithVersion(serviceExportVersion), meta.RESTScopeNamespace) - mapper.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) - return fake.NewClientBuilder(). - WithScheme(scheme). - WithRESTMapper(mapper). - WithStatusSubresource(&v1beta1.DynamoGraphDeployment{}). - WithInterceptorFuncs(funcs). - WithObjects(objects...). - Build() -} - -// listServiceExports returns the ServiceExports in namespace, read at version. -func listServiceExports(t *testing.T, c client.Client, version, namespace string) []unstructured.Unstructured { - t.Helper() - list := &unstructured.UnstructuredList{} - list.SetGroupVersionKind(multicluster.ServiceExportGroupKind.WithVersion(version).GroupVersion().WithKind("ServiceExportList")) - if err := c.List(context.Background(), list, client.InNamespace(namespace)); err != nil { - t.Fatalf("list ServiceExports: %v", err) - } - return list.Items -} - -// newPoolTestDGD returns a DGD named name in namespace dynamo with a decode -// worker and a frontend component for each of frontends. -func newPoolTestDGD(name string, frontends ...string) *v1beta1.DynamoGraphDeployment { - dgd := &v1beta1.DynamoGraphDeployment{Name: name, Namespace: "dynamo", UID: uuid.NewUUID()} - dgd.Spec.Components = []v1beta1.DynamoComponentDeploymentSharedSpec{ - {ComponentName: "VllmDecodeWorker", ComponentType: v1beta1.ComponentTypeDecode}, - } - for _, frontend := range frontends { - dgd.Spec.Components = append(dgd.Spec.Components, v1beta1.DynamoComponentDeploymentSharedSpec{ - ComponentName: frontend, ComponentType: v1beta1.ComponentTypeFrontend, - }) - } - return dgd -} - -// newPoolTestRelay returns the DynamoPoolRelay that makes dgd a pool of cluster -// east. -func newPoolTestRelay(dgd *v1beta1.DynamoGraphDeployment) *v1alpha1.DynamoPoolRelay { - return &v1alpha1.DynamoPoolRelay{ - Name: dgd.Name, Namespace: dgd.Namespace, - Spec: v1alpha1.DynamoPoolRelaySpec{ClusterID: "east"}, - } -} - -// ownedServiceExport returns the export at version of the Service named name -// that dgd controls. -func ownedServiceExport( - t *testing.T, - scheme *runtime.Scheme, - version string, - dgd *v1beta1.DynamoGraphDeployment, - name string, -) *unstructured.Unstructured { - t.Helper() - export := multicluster.RenderServiceExport(version, dgd.Namespace, name, dgd.Name) - if err := controllerutil.SetControllerReference(dgd, export, scheme); err != nil { - t.Fatalf("set the owner of ServiceExport %s: %v", name, err) - } - return export -} - -// withExportConditions sets the conditions an MCS implementation writes on export. -func withExportConditions(export *unstructured.Unstructured, conditions ...metav1.Condition) *unstructured.Unstructured { - items := make([]any, 0, len(conditions)) - for _, condition := range conditions { - items = append(items, map[string]any{ - "type": condition.Type, "status": string(condition.Status), - "reason": condition.Reason, "message": condition.Message, - }) - } - export.Object["status"] = map[string]any{"conditions": items} - return export -} - -// imported is the condition a standard implementation writes on an export it has -// imported. -var imported = metav1.Condition{Type: "Ready", Status: metav1.ConditionTrue, Reason: "Exported"} - -// newServiceExportTestReconciler returns the export reconciler with the gates -// that implementation resolves to in a cluster serving ServiceExports at version, -// reading exports through c, in a cluster with clusterSetID as its ID in its -// ClusterSet. -func newServiceExportTestReconciler( - c client.Client, - recorder events.EventRecorder, - implementation configv1alpha1.MCSImplementation, - version string, - clusterSetID string, -) *dgdServiceExportReconciler { - config := &configv1alpha1.OperatorConfiguration{ - MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: implementation}, - } - gates := features.Gates{MultiClusterExport: implementation != configv1alpha1.MCSImplementationNone} - if gates.MultiClusterExport { - gates.ServiceExportVersion = version - } - return newDGDServiceExportReconciler(newDGDResourceSyncer(c, recorder), config, gates, c, - func(context.Context) string { return clusterSetID }) -} - -func TestDGDServiceExportReconciler(t *testing.T) { - // -pool-relay is 56 characters, one more than Karmada imports; - // -frontend is 54. - longName := strings.Repeat("a", 45) - - for _, tc := range []struct { - name string - dgdName string - frontends []string - relay bool - relayTerminating bool - implementation configv1alpha1.MCSImplementation - ownedBefore []string - terminating bool - foreign []string - // ownedByAnother are exports another DGD controls. - ownedByAnother []string - wantOwned []string - wantEvents int - // wantReason is the condition's reason once the exports are in place, or - // empty when the DGD gets no condition. The fake client runs no MCS - // implementation, so a standard implementation never reports an import. - wantReason string - }{ - { - name: "a pool exports its PoolRelay and frontend Services", - dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{"qwen-pool-relay", "qwen-frontend"}, - wantEvents: 2, - wantReason: reasonPoolExported, - }, - { - name: "a standard implementation is waited for", - dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationStandard, - wantOwned: []string{"qwen-pool-relay", "qwen-frontend"}, - wantEvents: 2, - wantReason: reasonPoolExportPending, - }, - { - name: "each frontend component's Service is exported under its own name", - dgdName: "qwen", frontends: []string{"Frontend", "Web.v2"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{"qwen-pool-relay", "qwen-frontend", "qwen-web-v2"}, - wantEvents: 3, - wantReason: reasonPoolExported, - }, - { - name: "a renamed frontend's export moves to its new Service", - dgdName: "qwen", frontends: []string{"Web"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, - wantOwned: []string{"qwen-pool-relay", "qwen-web"}, - wantEvents: 2, - wantReason: reasonPoolExported, - }, - { - name: "a pool without a frontend exports its PoolRelay Service and reports it", - dgdName: "qwen", relay: true, - implementation: configv1alpha1.MCSImplementationStandard, - wantOwned: []string{"qwen-pool-relay"}, - wantEvents: 1, - wantReason: reasonPoolNoFrontend, - }, - { - name: "a DGD with a dot in its name has no PoolRelay Service to export", - dgdName: "qwen3-0.6b", frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{"qwen3-0-6b-frontend"}, - wantEvents: 1, - wantReason: reasonPoolInvalidServiceName, - }, - { - name: "deleting the DynamoPoolRelay withdraws the exports", - dgdName: "qwen", frontends: []string{"Frontend"}, - implementation: configv1alpha1.MCSImplementationKarmada, - ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, - wantEvents: 2, - }, - { - name: "a DynamoPoolRelay being deleted withdraws the exports", - dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, relayTerminating: true, - implementation: configv1alpha1.MCSImplementationStandard, - ownedBefore: []string{"qwen-pool-relay", "qwen-frontend"}, - wantEvents: 2, - }, - { - name: "an export already being deleted is not deleted again", - dgdName: "qwen", frontends: []string{"Frontend"}, - implementation: configv1alpha1.MCSImplementationStandard, - ownedBefore: []string{"qwen-pool-relay"}, - terminating: true, - wantOwned: []string{"qwen-pool-relay"}, - }, - { - name: "an export a user created is neither adopted nor deleted", - dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - foreign: []string{"qwen-frontend"}, - wantOwned: []string{"qwen-pool-relay"}, - wantEvents: 1, - wantReason: reasonPoolExported, - }, - { - name: "a DGD that is not a pool leaves a user's export alone", - dgdName: "qwen", frontends: []string{"Frontend"}, - implementation: configv1alpha1.MCSImplementationStandard, - foreign: []string{"qwen-pool-relay"}, - }, - { - name: "a DGD that is not a pool leaves another DGD's export alone", - dgdName: "qwen", frontends: []string{"Frontend"}, - implementation: configv1alpha1.MCSImplementationStandard, - ownedByAnother: []string{"qwen-frontend"}, - }, - { - name: "without an MCS implementation nothing is exported", - dgdName: "qwen", frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationNone, - }, - { - name: "Karmada skips a Service name it cannot import", - dgdName: longName, frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{longName + "-frontend"}, - wantEvents: 1, - wantReason: reasonPoolServiceNameTooLong, - }, - { - name: "standard MCS exports the same long names", - dgdName: longName, frontends: []string{"Frontend"}, relay: true, - implementation: configv1alpha1.MCSImplementationStandard, - wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, - wantEvents: 2, - wantReason: reasonPoolExportPending, - }, - } { - t.Run(tc.name, func(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - - t.Log("Create the DGD, its DynamoPoolRelay if it is a pool, and the exports that exist before reconciling") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD(tc.dgdName, tc.frontends...) - objects := []client.Object{dgd} - if tc.relay { - relay := newPoolTestRelay(dgd) - if tc.relayTerminating { - relay.Finalizers = []string{"example.com/hold"} - relay.DeletionTimestamp = &metav1.Time{Time: time.Now()} - } - objects = append(objects, relay) - } - for _, name := range tc.ownedBefore { - owned := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, name) - if tc.terminating { - owned.SetFinalizers([]string{"example.com/hold"}) - owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) - } - objects = append(objects, owned) - } - for _, name := range tc.foreign { - objects = append(objects, multicluster.RenderServiceExport(serviceExportTestVersion, dgd.Namespace, name, dgd.Name)) - } - another := newPoolTestDGD("another", "Frontend") - for _, name := range tc.ownedByAnother { - objects = append(objects, ownedServiceExport(t, scheme, serviceExportTestVersion, another, name)) - } - fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, objects...) - - t.Log("Reconcile the exports, with the gate the configured implementation resolves to") - recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, serviceExportTestVersion, "") - _, err := reconciler.Reconcile(ctx, dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) - - t.Log("Reconcile again: every export is in place, so nothing more is recorded, and the condition settles") - condition, err := reconciler.Reconcile(ctx, dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(recorder.Events).To(gomega.HaveLen(tc.wantEvents)) - if tc.wantReason == "" { - g.Expect(condition).To(gomega.BeNil()) - } else { - g.Expect(condition).NotTo(gomega.BeNil()) - g.Expect(condition.Type).To(gomega.Equal(globalRouterPoolExportedConditionType)) - g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason), condition.Message) - } - - t.Log("Check which exports the DGD controls, and that other exports are untouched") - var owned, others []string - for i, export := range listServiceExports(t, fakeClient, serviceExportTestVersion, dgd.Namespace) { - if !metav1.IsControlledBy(&export, dgd) { - others = append(others, export.GetName()) - continue - } - owned = append(owned, export.GetName()) - g.Expect(export.GetLabels()).To(gomega.HaveKeyWithValue(consts.KubeLabelDynamoGraphDeploymentName, dgd.Name), i) - } - g.Expect(owned).To(gomega.ConsistOf(tc.wantOwned)) - g.Expect(others).To(gomega.ConsistOf(append(append([]string{}, tc.foreign...), tc.ownedByAnother...))) - }) - } -} - -func TestDGDServiceExportReconcilerUsesTheResolvedVersion(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - const version = "v1beta1" - - t.Log("Serve ServiceExports at a version other than the tests' default, with an export to withdraw") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD("qwen", "Web") - fakeClient := newServiceExportTestClient(scheme, version, interceptor.Funcs{}, - dgd, newPoolTestRelay(dgd), ownedServiceExport(t, scheme, version, dgd, "qwen-frontend")) - - t.Log("Reconcile with the gates resolved to that version: exports are listed, created and withdrawn at it") - recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, version, "") - _, err := reconciler.Reconcile(ctx, dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(recorder.Events).To(gomega.HaveLen(3)) - var names []string - for _, export := range listServiceExports(t, fakeClient, version, dgd.Namespace) { - g.Expect(export.GetAPIVersion()).To(gomega.Equal("multicluster.x-k8s.io/" + version)) - names = append(names, export.GetName()) - } - g.Expect(names).To(gomega.ConsistOf("qwen-pool-relay", "qwen-web")) -} - -func TestDGDServiceExportReconcilerReportsTheImport(t *testing.T) { - for _, tc := range []struct { - name string - implementation configv1alpha1.MCSImplementation - relayConditions []metav1.Condition - frontendTerminating bool - wantStatus metav1.ConditionStatus - wantReason string - wantMessage string - }{ - { - name: "Karmada reports nothing", - implementation: configv1alpha1.MCSImplementationKarmada, - wantStatus: metav1.ConditionTrue, - wantReason: reasonPoolExported, - wantMessage: "Exported qwen-pool-relay, qwen-frontend through karmada", - }, - { - name: "a standard implementation imported both", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{imported}, - wantStatus: metav1.ConditionTrue, - wantReason: reasonPoolExported, - wantMessage: "Exported qwen-pool-relay, qwen-frontend through mcs", - }, - { - name: "a standard implementation has not reported yet", - implementation: configv1alpha1.MCSImplementationStandard, - wantStatus: metav1.ConditionUnknown, - wantReason: reasonPoolExportPending, - wantMessage: "Waiting for mcs to import qwen-pool-relay, qwen-frontend", - }, - { - name: "the MCS implementation finds no Service to export", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{ - {Type: "Valid", Status: metav1.ConditionFalse, Reason: "NoService", Message: "Service not found"}, - }, - wantStatus: metav1.ConditionFalse, - wantReason: reasonPoolExportRejected, - wantMessage: "mcs reports on the export of Service qwen-pool-relay: Valid is False (NoService): Service not found", - }, - { - name: "the import is pending", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{ - {Type: "Valid", Status: metav1.ConditionTrue}, {Type: "Ready", Status: metav1.ConditionFalse, Reason: "Pending"}, - }, - wantStatus: metav1.ConditionUnknown, - wantReason: reasonPoolExportPending, - wantMessage: "Waiting for mcs to import qwen-pool-relay", - }, - { - name: "the import failed", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{ - {Type: "Ready", Status: metav1.ConditionFalse, Reason: "Failed", Message: "refused"}, - }, - wantStatus: metav1.ConditionFalse, - wantReason: reasonPoolExportRejected, - wantMessage: "mcs reports on the export of Service qwen-pool-relay: Ready is False (Failed): refused", - }, - { - name: "the implementation resolved a conflict", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{ - imported, {Type: "Conflict", Status: metav1.ConditionTrue, Reason: "PortConflict", Message: "ports differ"}, - }, - wantStatus: metav1.ConditionTrue, - wantReason: reasonPoolExportConflict, - wantMessage: "Exported qwen-pool-relay, qwen-frontend through mcs, which resolves a conflict with another " + - "cluster's export of qwen-pool-relay (Conflict is True (PortConflict): ports differ)", - }, - { - name: "an export waits to be replaced", - implementation: configv1alpha1.MCSImplementationStandard, - relayConditions: []metav1.Condition{imported}, - frontendTerminating: true, - wantStatus: metav1.ConditionUnknown, - wantReason: reasonPoolExportPending, - wantMessage: "Waiting for mcs to import qwen-frontend", - }, - } { - t.Run(tc.name, func(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - t.Log("Create a pool whose exports exist, with what the MCS implementation reported on them") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD("qwen", "Frontend") - relayExport := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-pool-relay") - frontendExport := ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-frontend") - if len(tc.relayConditions) > 0 { - withExportConditions(relayExport, tc.relayConditions...) - withExportConditions(frontendExport, imported) - } - if tc.frontendTerminating { - frontendExport.SetFinalizers([]string{"example.com/hold"}) - frontendExport.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) - } - fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, - dgd, newPoolTestRelay(dgd), relayExport, frontendExport) - - t.Log("Reconcile and read the pool's condition") - recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, tc.implementation, serviceExportTestVersion, "") - condition, err := reconciler.Reconcile(context.Background(), dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(recorder.Events).To(gomega.BeEmpty()) - g.Expect(condition.Status).To(gomega.Equal(tc.wantStatus)) - g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason)) - g.Expect(condition.Message).To(gomega.Equal(tc.wantMessage)) - }) - } -} - -func TestDGDServiceExportReconcilerChecksTheClusterID(t *testing.T) { - for _, tc := range []struct { - name string - implementation configv1alpha1.MCSImplementation - // clusterSetID is the cluster's ID in its ClusterSet, or empty when unknown. - clusterSetID string - noClusterSetID bool - wantReason string - }{ - { - name: "a DynamoPoolRelay with the cluster's ID", - implementation: configv1alpha1.MCSImplementationStandard, - clusterSetID: "east", - wantReason: reasonPoolExported, - }, - { - name: "a DynamoPoolRelay with another cluster ID", - implementation: configv1alpha1.MCSImplementationStandard, - clusterSetID: "west", - wantReason: reasonPoolClusterIDMismatch, - }, - { - name: "a cluster whose ID in its ClusterSet is unknown", - implementation: configv1alpha1.MCSImplementationStandard, - wantReason: reasonPoolExported, - }, - { - name: "Karmada names member clusters itself", - implementation: configv1alpha1.MCSImplementationKarmada, - clusterSetID: "west", - wantReason: reasonPoolExported, - }, - { - name: "without a way to read the ID the check is skipped", - implementation: configv1alpha1.MCSImplementationStandard, - noClusterSetID: true, - wantReason: reasonPoolExported, - }, - } { - t.Run(tc.name, func(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - t.Log("Create a pool of cluster east whose exports are imported") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD("qwen", "Frontend") - fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, - dgd, newPoolTestRelay(dgd), - withExportConditions(ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-pool-relay"), imported), - withExportConditions(ownedServiceExport(t, scheme, serviceExportTestVersion, dgd, "qwen-frontend"), imported)) - - t.Log("Reconcile in a cluster with the given ID in its ClusterSet, and read the pool's condition") - reconciler := newServiceExportTestReconciler(fakeClient, events.NewFakeRecorder(10), tc.implementation, serviceExportTestVersion, tc.clusterSetID) - if tc.noClusterSetID { - reconciler.clusterSetID = nil - } - condition, err := reconciler.Reconcile(context.Background(), dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(condition.Reason).To(gomega.Equal(tc.wantReason), condition.Message) - if tc.wantReason == reasonPoolClusterIDMismatch { - g.Expect(condition.Status).To(gomega.Equal(metav1.ConditionFalse)) - g.Expect(condition.Message).To(gomega.ContainSubstring(`clusterId "east"`)) - g.Expect(condition.Message).To(gomega.ContainSubstring(`the ID "west"`)) - } - }) - } -} - -func TestClusterSetIDs(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - - t.Log("Serve the cluster's ID in its ClusterSet, counting reads and failing them on demand") - property := &unstructured.Unstructured{Object: map[string]any{"spec": map[string]any{"value": "east"}}} - property.SetGroupVersionKind(multicluster.ClusterPropertyGVK) - property.SetName(multicluster.ClusterIDProperty) - reads := 0 - var failure error - reader := newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), serviceExportTestVersion, interceptor.Funcs{ - Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { - reads++ - if failure != nil { - return failure - } - return c.Get(ctx, key, obj, opts...) - }, - }, property) - ids := &clusterSetIDs{} - start := time.Now() - - t.Log("The first call reads the ID, and calls within the TTL reuse it") - g.Expect(ids.get(ctx, reader, start)).To(gomega.Equal("east")) - g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL-time.Second))).To(gomega.Equal("east")) - g.Expect(reads).To(gomega.Equal(1)) - - t.Log("After the TTL a failed read skips the check and is not kept") - failure = errors.New("connection refused") - g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL))).To(gomega.BeEmpty()) - failure = nil - g.Expect(ids.get(ctx, reader, start.Add(clusterSetIDTTL))).To(gomega.Equal("east")) - g.Expect(reads).To(gomega.Equal(3)) -} - -func TestDGDServiceExportReconcilerToleratesAStaleCache(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - - t.Log("The cache has not yet seen exports that the API server already has") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD("qwen", "Frontend") - fakeClient := newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ - Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { - return apierrors.NewAlreadyExists(schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, obj.GetName()) - }, - }, dgd, newPoolTestRelay(dgd)) - - t.Log("Reconcile: AlreadyExists is no error, nothing is announced, and the exports are pending") - recorder := events.NewFakeRecorder(10) - reconciler := newServiceExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, serviceExportTestVersion, "") - condition, err := reconciler.Reconcile(ctx, dgd) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(recorder.Events).To(gomega.BeEmpty()) - g.Expect(condition.Reason).To(gomega.Equal(reasonPoolExportPending)) -} - -func TestMapServiceExportToDGDRequests(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - - t.Log("Create two DGDs in one namespace, one with a dot in its name, and a third elsewhere") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - llama := newPoolTestDGD("llama", "Frontend", "Web") - dotted := newPoolTestDGD("qwen3-0.6b", "Frontend") - elsewhere := newPoolTestDGD("llama", "Frontend") - elsewhere.Namespace = "other" - reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, llama, dotted, elsewhere), - } - - for _, tc := range []struct { - name string - export *unstructured.Unstructured - want []ctrl.Request - }{ - { - name: "a PoolRelay Service", - export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama-pool-relay", "llama"), - want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, - }, - { - name: "a frontend Service under its component's name", - export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama-web", "llama"), - want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, - }, - { - name: "the frontend Service of a DGD with a dot in its name", - export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "qwen3-0-6b-frontend", "qwen3-0.6b"), - want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(dotted)}}, - }, - { - name: "an export the DGD controls for a frontend it no longer has", - export: ownedServiceExport(t, scheme, serviceExportTestVersion, llama, "llama-old"), - want: []ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(llama)}}, - }, - { - name: "another Service", - export: multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "llama", "llama"), - want: []ctrl.Request{}, - }, - } { - t.Log("Map an event for " + tc.name + " in namespace dynamo to the DGDs it concerns") - g.Expect(reconciler.mapServiceExportToDGDRequests(ctx, tc.export)).To(gomega.Equal(tc.want), tc.name) - } -} - -func TestPoolWatchPredicates(t *testing.T) { - g := gomega.NewGomegaWithT(t) - deleting := &metav1.Time{Time: time.Now()} - - t.Log("A DynamoPoolRelay event requeues the DGD of the same name") - relay := newPoolTestRelay(newPoolTestDGD("qwen")) - g.Expect(mapPoolRelayToDGDRequests(context.Background(), relay)). - To(gomega.Equal([]ctrl.Request{{NamespacedName: client.ObjectKeyFromObject(relay)}})) - - t.Log("A DynamoPoolRelay update counts when its spec changes or its deletion starts") - respecified := relay.DeepCopy() - respecified.Generation = 2 - relabeled := relay.DeepCopy() - relabeled.Labels = map[string]string{"team": "router"} - withdrawn := relay.DeepCopy() - withdrawn.DeletionTimestamp = deleting - g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: respecified})).To(gomega.BeTrue()) - g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: withdrawn})).To(gomega.BeTrue()) - g.Expect(poolRelayChanged(event.UpdateEvent{ObjectOld: relay, ObjectNew: relabeled})).To(gomega.BeFalse()) - - t.Log("A ServiceExport update counts when the MCS implementation reports or its deletion starts") - export := multicluster.RenderServiceExport(serviceExportTestVersion, "dynamo", "qwen-frontend", "qwen") - reported := withExportConditions(export.DeepCopy(), imported) - relabeledExport := export.DeepCopy() - relabeledExport.SetLabels(map[string]string{"team": "router"}) - deletingExport := export.DeepCopy() - deletingExport.SetDeletionTimestamp(deleting) - g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: reported})).To(gomega.BeTrue()) - g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: deletingExport})).To(gomega.BeTrue()) - g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: export, ObjectNew: relabeledExport})).To(gomega.BeFalse()) - g.Expect(serviceExportStateChanged(event.UpdateEvent{ObjectOld: &corev1.Service{}, ObjectNew: &corev1.Service{}})).To(gomega.BeFalse()) -} - -func TestApplyPoolExportCondition(t *testing.T) { - condition := func(status metav1.ConditionStatus, reason string) *metav1.Condition { - return &metav1.Condition{ - Type: globalRouterPoolExportedConditionType, Status: status, Reason: reason, Message: reason + " message", - } - } - for _, tc := range []struct { - name string - previous *metav1.Condition - condition *metav1.Condition - wantEvents []string - }{ - { - name: "a new problem is set and announced", - condition: condition(metav1.ConditionFalse, reasonPoolNoFrontend), - wantEvents: []string{reasonPoolNoFrontend}, - }, - { - name: "a problem already reported is not announced again", - previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), - condition: condition(metav1.ConditionFalse, reasonPoolNoFrontend), - }, - { - name: "another problem is announced", - previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), - condition: condition(metav1.ConditionFalse, reasonPoolClusterIDMismatch), - wantEvents: []string{reasonPoolClusterIDMismatch}, - }, - { - name: "a conflict the implementation resolved is announced", - previous: condition(metav1.ConditionTrue, reasonPoolExported), - condition: condition(metav1.ConditionTrue, reasonPoolExportConflict), - wantEvents: []string{reasonPoolExportConflict}, - }, - { - name: "a failed reconcile records its own event", - condition: condition(metav1.ConditionFalse, reasonPoolExportFailed), - }, - { - name: "a pending import is not announced", - condition: condition(metav1.ConditionUnknown, reasonPoolExportPending), - }, - { - name: "an export that works is not announced", - previous: condition(metav1.ConditionFalse, reasonPoolNoFrontend), - condition: condition(metav1.ConditionTrue, reasonPoolExported), - }, - { - name: "a DGD that is no longer a pool loses the condition", - previous: condition(metav1.ConditionTrue, reasonPoolExported), - }, - } { - t.Run(tc.name, func(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - t.Log("Start from the status the DGD has, as the workload program does") - dgd := newPoolTestDGD("qwen", "Frontend") - if tc.previous != nil { - meta.SetStatusCondition(&dgd.Status.Conditions, *tc.previous) - } - result := newWorkloadProgramResult(dgd) - - t.Log("Apply the pool's condition") - applyPoolExportCondition(dgd, &result, tc.condition) - applied := meta.FindStatusCondition(result.Status.Conditions, globalRouterPoolExportedConditionType) - if tc.condition == nil { - g.Expect(applied).To(gomega.BeNil()) - } else { - g.Expect(applied).NotTo(gomega.BeNil()) - g.Expect(applied.Status).To(gomega.Equal(tc.condition.Status)) - g.Expect(applied.Reason).To(gomega.Equal(tc.condition.Reason)) - } - reasons := make([]string, 0, len(result.Events)) - for _, recorded := range result.Events { - g.Expect(recorded.Type).To(gomega.Equal(corev1.EventTypeWarning)) - reasons = append(reasons, recorded.Reason) - } - g.Expect(reasons).To(gomega.ConsistOf(tc.wantEvents)) - }) - } -} - -func TestReconcilePoolExportsRetriesWithoutFailingTheDGD(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := context.Background() - - t.Log("Make a DGD a pool, and refuse every ServiceExport create, as an admission policy would") - scheme := newDynamoGraphDeploymentControllerTestScheme(t) - dgd := newPoolTestDGD("llama", "Frontend") - relay := newPoolTestRelay(dgd) - refused := interceptor.Funcs{ - Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { - return apierrors.NewForbidden( - schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, - "llama-pool-relay", errors.New("denied by policy")) - }, - } - recorder := events.NewFakeRecorder(10) - reconciler := &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(scheme, serviceExportTestVersion, refused, dgd, relay), - Recorder: recorder, - Config: &configv1alpha1.OperatorConfiguration{ - MultiCluster: configv1alpha1.MultiClusterConfiguration{ - MCSImplementation: configv1alpha1.MCSImplementationKarmada, - }, - }, - RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: features.Gates{ - MultiClusterExport: true, ServiceExportVersion: serviceExportTestVersion, - }}, - } - - t.Log("The failure asks for a retry, is recorded as a warning and in the condition, and returns no error that would fail the DGD") - result := reconciler.reconcilePoolExports(ctx, dgd) - g.Expect(result.retry).To(gomega.BeTrue()) - g.Expect(result.condition.Status).To(gomega.Equal(metav1.ConditionFalse)) - g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExportFailed)) - g.Expect(recorder.Events).To(gomega.HaveLen(1)) - g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) - - t.Log("A failure to read the DynamoPoolRelay leaves the condition as it was, since the DGD may not be a pool") - meta.SetStatusCondition(&dgd.Status.Conditions, *newPoolCondition(dgd, metav1.ConditionTrue, reasonPoolExported, "exported")) - reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ - Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { - if _, ok := obj.(*v1alpha1.DynamoPoolRelay); ok { - return errors.New("cache not synced") - } - return c.Get(ctx, key, obj, opts...) - }, - }, dgd, relay) - result = reconciler.reconcilePoolExports(ctx, dgd) - g.Expect(result.retry).To(gomega.BeTrue()) - g.Expect(result.condition).NotTo(gomega.BeNil()) - g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExported)) - g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) - - t.Log("A failed withdrawal from a DGD that is no longer a pool keeps the condition until the exports are gone") - former := newPoolTestDGD("qwen", "Frontend") - meta.SetStatusCondition(&former.Status.Conditions, *newPoolCondition(former, metav1.ConditionTrue, reasonPoolExported, "exported")) - reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{ - Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { - return errors.New("denied by policy") - }, - }, former, ownedServiceExport(t, scheme, serviceExportTestVersion, former, "qwen-frontend")) - result = reconciler.reconcilePoolExports(ctx, former) - g.Expect(result.retry).To(gomega.BeTrue()) - g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExported)) - g.Expect(<-recorder.Events).To(gomega.HavePrefix(corev1.EventTypeWarning + " PoolExportFailed")) - - t.Log("Exports that succeed ask for no retry") - reconciler.Client = newServiceExportTestClient(scheme, serviceExportTestVersion, interceptor.Funcs{}, dgd, relay) - result = reconciler.reconcilePoolExports(ctx, dgd) - g.Expect(result.retry).To(gomega.BeFalse()) - g.Expect(result.condition.Reason).To(gomega.Equal(reasonPoolExportPending)) -} - -// newPoolTestDGDReconciler returns a DGD reconciler over objects, with standard -// MCS and the gates given. -func newPoolTestDGDReconciler( - t *testing.T, - gates features.Gates, - objects ...client.Object, -) (*DynamoGraphDeploymentReconciler, *events.FakeRecorder) { - gates.MultiClusterExport, gates.ServiceExportVersion = true, serviceExportTestVersion - recorder := events.NewFakeRecorder(10) - return &DynamoGraphDeploymentReconciler{ - Client: newServiceExportTestClient(newDynamoGraphDeploymentControllerTestScheme(t), serviceExportTestVersion, interceptor.Funcs{}, objects...), - Recorder: recorder, - Config: &configv1alpha1.OperatorConfiguration{ - MultiCluster: configv1alpha1.MultiClusterConfiguration{ - MCSImplementation: configv1alpha1.MCSImplementationStandard, - }, - }, - RuntimeConfig: &commoncontroller.RuntimeConfig{Gate: gates}, - DockerSecretRetriever: &mockDockerSecretRetriever{ - GetSecretsFunc: func(namespace, imageName string) ([]string, error) { return nil, nil }, - }, - }, recorder -} - -// drainEvents returns the events recorder holds. -func drainEvents(recorder *events.FakeRecorder) []string { - var recorded []string - for len(recorder.Events) > 0 { - recorded = append(recorded, <-recorder.Events) - } - return recorded -} - -func TestDynamoGraphDeploymentReconcilePersistsThePoolCondition(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := t.Context() - - t.Log("Create a finalized DGD on the component provider, whose program rejects its only component, and its DynamoPoolRelay") - dgd := &v1beta1.DynamoGraphDeployment{ - Name: "llama", Namespace: "dynamo", Generation: 4, UID: uuid.NewUUID(), - Annotations: map[string]string{consts.KubeAnnotationWorkloadProvider: consts.WorkloadProviderComponent}, - Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ - {ComponentName: "serving", ComponentType: v1beta1.ComponentTypeLPX}, - }}, - } - commoncontroller.AddFinalizer(dgd) - relay := newPoolTestRelay(dgd) - reconciler, recorder := newPoolTestDGDReconciler(t, features.Gates{LPX: true}, dgd, relay) - reconcileAndRead := func() (*v1beta1.DynamoGraphDeployment, []string) { - _, err := reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) - g.Expect(errors.Is(err, reconcile.TerminalError(nil))).To(gomega.BeTrue(), "%v", err) - stored := &v1beta1.DynamoGraphDeployment{} - g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) - return stored, drainEvents(recorder) - } - noFrontend := gomega.HavePrefix(corev1.EventTypeWarning + " " + reasonPoolNoFrontend) - - t.Log("Reconcile: the program's status is persisted with the pool's condition, and the problem is announced") - stored, recorded := reconcileAndRead() - ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready") - g.Expect(ready).NotTo(gomega.BeNil()) - g.Expect(ready.Reason).To(gomega.Equal("UnsupportedComponent")) - pool := meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType) - g.Expect(pool).NotTo(gomega.BeNil()) - g.Expect(pool.Status).To(gomega.Equal(metav1.ConditionFalse)) - g.Expect(pool.Reason).To(gomega.Equal(reasonPoolNoFrontend)) - g.Expect(pool.ObservedGeneration).To(gomega.Equal(dgd.Generation)) - g.Expect(recorded).To(gomega.ContainElement(noFrontend)) - - t.Log("Reconcile again: the same problem is not announced twice") - _, recorded = reconcileAndRead() - g.Expect(recorded).NotTo(gomega.ContainElement(noFrontend)) - - t.Log("Delete the DynamoPoolRelay: the export is withdrawn and the condition removed") - g.Expect(reconciler.Delete(ctx, relay)).To(gomega.Succeed()) - stored, _ = reconcileAndRead() - g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType)).To(gomega.BeNil()) - g.Expect(listServiceExports(t, reconciler.Client, serviceExportTestVersion, dgd.Namespace)).To(gomega.BeEmpty()) -} - -func TestDynamoGraphDeploymentReconcileKeepsThePoolConditionOnEarlyFailures(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ctx := t.Context() - - t.Log("Store a DGD with an LPX component while LPX is disabled, and its DynamoPoolRelay") - dgd := &v1beta1.DynamoGraphDeployment{ - Name: "mixed", Namespace: "dynamo", Generation: 4, UID: uuid.NewUUID(), - Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ - {ComponentName: "worker", ComponentType: v1beta1.ComponentTypeLPX}, - }}, - } - relay := newPoolTestRelay(dgd) - reconciler, _ := newPoolTestDGDReconciler(t, features.Gates{}, dgd, relay) - - t.Log("Reconcile: the early failure's status carries the pool's condition") - _, err := reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) - g.Expect(err).NotTo(gomega.HaveOccurred()) - stored := &v1beta1.DynamoGraphDeployment{} - g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) - g.Expect(meta.FindStatusCondition(stored.Status.Conditions, "Ready").Reason).To(gomega.Equal("lpx_disabled")) - g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType).Reason). - To(gomega.Equal(reasonPoolNoFrontend)) - - t.Log("Delete the DynamoPoolRelay: the next early failure removes the condition") - g.Expect(reconciler.Delete(ctx, relay)).To(gomega.Succeed()) - _, err = reconciler.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)}) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(dgd), stored)).To(gomega.Succeed()) - g.Expect(meta.FindStatusCondition(stored.Status.Conditions, globalRouterPoolExportedConditionType)).To(gomega.BeNil()) -} - -func TestRetryPoolExports(t *testing.T) { - g := gomega.NewGomegaWithT(t) - for _, tc := range []struct { - name string - result ctrl.Result - err error - want ctrl.Result - }{ - {name: "a reconcile that would not run again retries the exports", want: ctrl.Result{RequeueAfter: poolExportRetry}}, - {name: "a later retry comes sooner", result: ctrl.Result{RequeueAfter: time.Hour}, want: ctrl.Result{RequeueAfter: poolExportRetry}}, - {name: "a sooner retry stays", result: ctrl.Result{RequeueAfter: time.Second}, want: ctrl.Result{RequeueAfter: time.Second}}, - {name: "an error is retried anyway", err: errors.New("failed"), want: ctrl.Result{}}, - } { - t.Log(tc.name) - g.Expect(retryPoolExports(tc.result, tc.err)).To(gomega.Equal(tc.want), tc.name) - } -} diff --git a/deploy/operator/internal/controller/dgd_workload_program_test.go b/deploy/operator/internal/controller/dgd_workload_program_test.go index 5470d24248a1..e8e71674e41a 100644 --- a/deploy/operator/internal/controller/dgd_workload_program_test.go +++ b/deploy/operator/internal/controller/dgd_workload_program_test.go @@ -256,7 +256,7 @@ func TestPersistWorkloadProgramResultEmitsEventsAfterStatusUpdate(t *testing.T) result.Eventf(corev1.EventTypeNormal, "Transition", "transition persisted") t.Log("Persist status through the outer controller boundary") - err := reconciler.persistWorkloadProgramResult(context.Background(), dgd, result, nil) + err := reconciler.persistWorkloadProgramResult(context.Background(), dgd, result) t.Log("Verify event publication is strictly ordered after successful status persistence") require.True(t, statusUpdated) diff --git a/deploy/operator/internal/controller/dynamographdeployment_controller.go b/deploy/operator/internal/controller/dynamographdeployment_controller.go index 1b2661e7c7cb..edfa9b17311f 100644 --- a/deploy/operator/internal/controller/dynamographdeployment_controller.go +++ b/deploy/operator/internal/controller/dynamographdeployment_controller.go @@ -21,9 +21,7 @@ import ( "context" "errors" "fmt" - "slices" "strings" - "time" "github.com/ai-dynamo/dynamo/deploy/operator/internal/secret" @@ -31,7 +29,6 @@ import ( corev1 "k8s.io/api/core/v1" resourcev1 "k8s.io/api/resource/v1" "k8s.io/apimachinery/pkg/api/meta" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/client-go/rest" "k8s.io/client-go/tools/events" @@ -54,7 +51,6 @@ import ( "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo/epp" "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" - "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" snapshotv1alpha1 "github.com/ai-dynamo/snapshot/api/v1alpha1" ) @@ -80,16 +76,6 @@ type DynamoGraphDeploymentReconciler struct { Recorder events.EventRecorder DockerSecretRetriever DockerSecretRetriever SSHKeyManager *secret.SSHKeyManager - // APIReader reads objects that the cache does not hold, such as the cluster's - // ID in its ClusterSet. - APIReader client.Reader - // Cache reads unstructured objects, such as ServiceExports, from the - // informers that the controller's watches start. The manager's client reads - // unstructured objects from the API server. - Cache client.Reader - // clusterSetIDs keeps the cluster's ID in its ClusterSet between reconciles. - // Nil skips the check of a pool's cluster ID. - clusterSetIDs *clusterSetIDs } // +kubebuilder:rbac:groups=nvidia.com,resources=dynamographdeployments,verbs=get;list;watch;create;update;patch;delete @@ -106,9 +92,6 @@ type DynamoGraphDeploymentReconciler struct { // +kubebuilder:rbac:groups=scheduling.run.ai,resources=queues,verbs=get;list // +kubebuilder:rbac:groups=inference.networking.k8s.io,resources=inferencepools,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=networking.istio.io,resources=destinationrules,verbs=get;list;watch;create;update;patch;delete -// +kubebuilder:rbac:groups=multicluster.x-k8s.io,resources=serviceexports,verbs=get;list;watch;create;delete -// +kubebuilder:rbac:groups=nvidia.com,resources=dynamopoolrelays,verbs=get;list;watch -// +kubebuilder:rbac:groups=about.k8s.io,resources=clusterproperties,verbs=get // +kubebuilder:rbac:groups=nvidia.com,resources=snapshotjobs,verbs=get;list;watch;create;patch;delete // +kubebuilder:rbac:groups=nvidia.com,resources=podsnapshots,verbs=get;list;watch;patch;delete // +kubebuilder:rbac:groups=resource.k8s.io,resources=resourceclaimtemplates,verbs=get;list;watch;create;update;patch;delete @@ -149,19 +132,11 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr return ctrl.Result{}, err } - // A Global Router pool's exports follow its DynamoPoolRelay whatever happens to - // its workloads below. A failed export never fails the DGD: it is logged, - // recorded as an event and in the pool's condition, and retried. - poolExports := r.reconcilePoolExports(ctx, dynamoDeployment) - if poolExports.retry { - defer func() { result = retryPoolExports(result, err) }() - } - // Reject stored LPX graphs before provider selection or any workload writes. if dynamoDeployment.HasLPXComponent() && !r.RuntimeConfig.Gate.Enabled(features.LPX) { programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, "lpx_disabled", errors.New("LPX integration is disabled")) - return programResult.Result, r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition) + return programResult.Result, r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult) } // Lock in the provider before allowing any other reconciliation effects. @@ -176,7 +151,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr // Persist the invalid selection diagnosis before suppressing automatic retries. programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, reasonUnsupportedWorkloadProvider, err) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { return programResult.Result, statusErr } return programResult.Result, reconcile.TerminalError(err) @@ -199,7 +174,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr if compatibilityErr := errors.Join(compatibilityErrs...); compatibilityErr != nil { programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, reasonFailedToReconcileResources, compatibilityErr) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { logger.Error(statusErr, "unable to update status after compatibility failure") return ctrl.Result{}, statusErr } @@ -211,7 +186,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr logger.Error(err, "failed to handle the finalizer") programResult := newWorkloadProgramResult(dynamoDeployment) programResult.Fail(dynamoDeployment.Generation, "failed_to_handle_the_finalizer", err) - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { logger.Error(statusErr, "unable to update status after finalizer failure") } return ctrl.Result{}, err @@ -237,7 +212,7 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr "Refusing to reconcile a resource with conflicting controller ownership: %s", ownershipConflictCondition.Message) } result = programResult.Result - if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult, poolExports.condition); statusErr != nil { + if statusErr := r.persistWorkloadProgramResult(ctx, dynamoDeployment, programResult); statusErr != nil { logger.Error(statusErr, "unable to persist workload program status") return result, statusErr } @@ -250,15 +225,11 @@ func (r *DynamoGraphDeploymentReconciler) Reconcile(ctx context.Context, req ctr return result, nil } -// persistWorkloadProgramResult writes result's status, with the Global Router -// pool's condition, and records result's events once the status is written. func (r *DynamoGraphDeploymentReconciler) persistWorkloadProgramResult( ctx context.Context, dgd *nvidiacomv1beta1.DynamoGraphDeployment, result workloadProgramResult, - poolCondition *metav1.Condition, ) error { - applyPoolExportCondition(dgd, &result, poolCondition) dgd.Status = result.Status if err := r.Status().Update(ctx, dgd); err != nil { return fmt.Errorf("update DynamoGraphDeployment status: %w", err) @@ -294,97 +265,6 @@ func workloadRoutingAnnotationsChanged(update event.UpdateEvent) bool { annotationValue(oldDGD, consts.KubeAnnotationEnableGrove) != annotationValue(newDGD, consts.KubeAnnotationEnableGrove) } -// poolExportRetry is how soon a DGD whose pool exports failed is reconciled again. -const poolExportRetry = 30 * time.Second - -// poolExportResult is the outcome of reconciling a Global Router pool's exports. -type poolExportResult struct { - // retry asks for the DGD to be reconciled again because the exports failed. - retry bool - // condition is the GlobalRouterPoolExported condition for the DGD, or nil to - // remove it. - condition *metav1.Condition -} - -// reconcilePoolExports converges a Global Router pool's ServiceExports. A failure -// is logged, recorded as a Warning event and, when the DGD is known to be a -// pool, in its condition, and retried. -func (r *DynamoGraphDeploymentReconciler) reconcilePoolExports( - ctx context.Context, - dgd *nvidiacomv1beta1.DynamoGraphDeployment, -) poolExportResult { - exports := newDGDServiceExportReconciler( - newDGDResourceSyncer(r.Client, r.Recorder), r.Config, r.RuntimeConfig.Gate, - r.serviceExportReader(), r.clusterSetID) - condition, err := exports.Reconcile(ctx, dgd) - if err == nil { - return poolExportResult{condition: condition} - } - log.FromContext(ctx).Error(err, "Failed to reconcile Global Router pool ServiceExports") - if r.Recorder != nil { - r.Recorder.Eventf(dgd, nil, corev1.EventTypeWarning, "PoolExportFailed", "Export", - "Failed to reconcile Global Router pool ServiceExports: %v", err) - } - // Whether the DGD is a pool is unknown, so its condition stays as it is. - if condition == nil { - if previous := meta.FindStatusCondition(dgd.Status.Conditions, globalRouterPoolExportedConditionType); previous != nil { - kept := *previous - condition = &kept - } - } - return poolExportResult{retry: true, condition: condition} -} - -// serviceExportReader returns the reader of ServiceExports: the cache, or the -// client when the controller has no cache, as in tests. -func (r *DynamoGraphDeploymentReconciler) serviceExportReader() client.Reader { - if r.Cache != nil { - return r.Cache - } - return r.Client -} - -// clusterSetID returns the cluster's ID in its ClusterSet, or "" when it is -// unknown. -func (r *DynamoGraphDeploymentReconciler) clusterSetID(ctx context.Context) string { - if r.APIReader == nil || r.clusterSetIDs == nil { - return "" - } - return r.clusterSetIDs.get(ctx, r.APIReader, time.Now()) -} - -// applyPoolExportCondition sets the GlobalRouterPoolExported condition in the -// status about to be persisted, or removes it when condition is nil. A problem -// that the condition newly reports is also recorded as a Warning event. -func applyPoolExportCondition( - dgd *nvidiacomv1beta1.DynamoGraphDeployment, - result *workloadProgramResult, - condition *metav1.Condition, -) { - if condition == nil { - meta.RemoveStatusCondition(&result.Status.Conditions, globalRouterPoolExportedConditionType) - return - } - previous := meta.FindStatusCondition(dgd.Status.Conditions, globalRouterPoolExportedConditionType) - meta.SetStatusCondition(&result.Status.Conditions, *condition) - raised := previous == nil || previous.Status != condition.Status || previous.Reason != condition.Reason - if raised && slices.Contains(poolProblemReasons, condition.Reason) { - result.Eventf(corev1.EventTypeWarning, condition.Reason, "%s", condition.Message) - } -} - -// retryPoolExports makes a reconcile run again within poolExportRetry, unless it -// returned an error, which is retried anyway, or already runs again sooner. -func retryPoolExports(result ctrl.Result, err error) ctrl.Result { - if err != nil { - return result - } - if result.RequeueAfter == 0 || result.RequeueAfter > poolExportRetry { - result.RequeueAfter = poolExportRetry - } - return result -} - // SetupWithManager sets up the controller with the Manager. func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) error { if err := mgr.GetFieldIndexer().IndexField( @@ -503,31 +383,6 @@ func (r *DynamoGraphDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) err GenericFunc: func(ge event.GenericEvent) bool { return false }, })) } - // A Global Router pool's exports converge once the cache sees them created or - // deleted, the DGD's own and a user's of the same Service alike, and its - // condition follows what the MCS implementation reports on them. A - // DynamoPoolRelay makes the DGD of the same name a pool. - if r.RuntimeConfig.Gate.Enabled(features.MultiClusterExport) { - ctrlBuilder = ctrlBuilder.Watches( - multicluster.NewServiceExport(r.RuntimeConfig.Gate.ServiceExportVersion), - handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToDGDRequests), - builder.WithPredicates(predicate.Funcs{ - CreateFunc: func(ce event.CreateEvent) bool { return true }, - DeleteFunc: func(de event.DeleteEvent) bool { return true }, - UpdateFunc: serviceExportStateChanged, - GenericFunc: func(ge event.GenericEvent) bool { return false }, - }), - ).Watches( - &nvidiacomv1alpha1.DynamoPoolRelay{}, - handler.EnqueueRequestsFromMapFunc(mapPoolRelayToDGDRequests), - builder.WithPredicates(predicate.Funcs{ - CreateFunc: func(ce event.CreateEvent) bool { return true }, - DeleteFunc: func(de event.DeleteEvent) bool { return true }, - UpdateFunc: poolRelayChanged, - GenericFunc: func(ge event.GenericEvent) bool { return false }, - }), - ) - } if r.RuntimeConfig.Gate.Enabled(features.DisaggregatedSet) { ctrlBuilder = newDisaggregatedSetWatchSetup(mgr.GetClient()).addTo(ctrlBuilder) } diff --git a/deploy/operator/internal/controller/poolrelay/controller.go b/deploy/operator/internal/controller/poolrelay/controller.go index 66cbdd0dc6b2..a8bbee49ea2d 100644 --- a/deploy/operator/internal/controller/poolrelay/controller.go +++ b/deploy/operator/internal/controller/poolrelay/controller.go @@ -23,6 +23,7 @@ import ( commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" "github.com/ai-dynamo/dynamo/deploy/operator/internal/dynamo" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" @@ -58,14 +59,18 @@ type reconciler struct { client.Client scheme *runtime.Scheme defaultBackend configv1alpha1.DiscoveryBackend + // exports configures the pool's ServiceExports, which the controller maintains + // when the multiClusterExport gate is on, and is nil otherwise. + exports *exportSettings } -// Setup registers the one-DGD Relay controller and its DGD dependency watch. +// Setup registers the one-DGD Relay controller and its DGD dependency watch, and, +// when the multiClusterExport gate is on, the watches of the pool's exports. func Setup(mgr ctrl.Manager, config *configv1alpha1.OperatorConfiguration, runtimeConfig *commoncontroller.RuntimeConfig) error { r := &reconciler{Client: mgr.GetClient(), scheme: mgr.GetScheme(), defaultBackend: config.Discovery.Backend} - return ctrl.NewControllerManagedBy(mgr). + controllerBuilder := ctrl.NewControllerManagedBy(mgr). For(&v1alpha1.DynamoPoolRelay{}). - WithEventFilter(commoncontroller.EphemeralDeploymentEventFilter(config, runtimeConfig)). + WithEventFilter(eventFilter(config, runtimeConfig)). Watches(&v1beta1.DynamoGraphDeployment{}, handler.EnqueueRequestsFromMapFunc(func(_ context.Context, object client.Object) []ctrl.Request { key := types.NamespacedName{Namespace: object.GetNamespace(), Name: object.GetName()} return []ctrl.Request{{NamespacedName: key}} @@ -75,8 +80,14 @@ func Setup(mgr ctrl.Manager, config *configv1alpha1.OperatorConfiguration, runti Owns(&corev1.ConfigMap{}). Owns(&corev1.ServiceAccount{}). Owns(&rbacv1.Role{}). - Owns(&rbacv1.RoleBinding{}). - Complete(r) + Owns(&rbacv1.RoleBinding{}) + if runtimeConfig.Gate.Enabled(features.MultiClusterExport) { + var err error + if controllerBuilder, err = r.watchExports(mgr, controllerBuilder, config, runtimeConfig); err != nil { + return err + } + } + return controllerBuilder.Complete(r) } // +kubebuilder:rbac:groups=nvidia.com,resources=dynamopoolrelays,verbs=get;list;watch @@ -89,7 +100,8 @@ func Setup(mgr ctrl.Manager, config *configv1alpha1.OperatorConfiguration, runti // +kubebuilder:rbac:groups=core,resources=configmaps;serviceaccounts,verbs=get;list;watch;create;update // +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=roles;rolebindings,verbs=get;list;watch;create;update -// Reconcile keeps one Relay Deployment and Service in step with its named DGD. +// Reconcile keeps one Relay Deployment and Service in step with its named DGD, and +// the pool's exports with them. func (r *reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl.Result, error) { var relay v1alpha1.DynamoPoolRelay if err := r.Get(ctx, request.NamespacedName, &relay); err != nil { @@ -140,22 +152,45 @@ func (r *reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl. if err != nil { return r.withdraw(ctx, &relay, "InvalidConfiguration", err.Error()) } + result, err := r.publish(ctx, &relay, resources, desiredDeployment) + if errors.Is(err, errOwnershipConflict) { + // publish withdrew the Relay, and with it the pool's exports. + return result, err + } + // The pool's exports follow the DGD whether or not the Relay's own resources + // applied: a failure of one does not hold back the other. + if err := errors.Join(err, r.reconcileExports(ctx, &relay, &dgd)); err != nil { + // An error requeues the Relay with backoff, which replaces the result. + return ctrl.Result{}, err + } + return result, nil +} + +// publish applies the Relay's resources and Deployment, and reports the +// Deployment's progress in the Relay's status. A resource that another owner +// controls withdraws the Relay, with an error that wraps errOwnershipConflict. +func (r *reconciler) publish( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + resources []client.Object, + desiredDeployment *appsv1.Deployment, +) (ctrl.Result, error) { for _, resource := range resources { - if err := r.apply(ctx, &relay, resource); err != nil { + if err := r.apply(ctx, relay, resource); err != nil { if errors.Is(err, errOwnershipConflict) { - _, withdrawErr := r.withdraw(ctx, &relay, "ResourceOwnershipConflict", err.Error()) + _, withdrawErr := r.withdraw(ctx, relay, "ResourceOwnershipConflict", err.Error()) return ctrl.Result{}, errors.Join(err, withdrawErr) } - return ctrl.Result{}, errors.Join(err, r.reportApplyFailure(ctx, &relay, err)) + return ctrl.Result{}, errors.Join(err, r.reportApplyFailure(ctx, relay, err)) } } - deployment, err := r.applyDeployment(ctx, &relay, desiredDeployment) + deployment, err := r.applyDeployment(ctx, relay, desiredDeployment) if err != nil { if errors.Is(err, errOwnershipConflict) { - _, withdrawErr := r.withdraw(ctx, &relay, "ResourceOwnershipConflict", err.Error()) + _, withdrawErr := r.withdraw(ctx, relay, "ResourceOwnershipConflict", err.Error()) return ctrl.Result{}, errors.Join(err, withdrawErr) } - return ctrl.Result{}, errors.Join(err, r.reportApplyFailure(ctx, &relay, err)) + return ctrl.Result{}, errors.Join(err, r.reportApplyFailure(ctx, relay, err)) } // Deployment status is listener availability, never model or KV readiness. @@ -168,7 +203,7 @@ func (r *reconciler) Reconcile(ctx context.Context, request ctrl.Request) (ctrl. if ready { reason, message = "ListenerAvailable", "All requested Relay listeners are available" } - if err := r.setStatus(ctx, &relay, ready, reason, message, serviceName(relay.Name), available, true); err != nil { + if err := r.setStatus(ctx, relay, ready, reason, message, serviceName(relay.Name), available, true); err != nil { return ctrl.Result{}, err } if !ready { @@ -261,6 +296,10 @@ func (r *reconciler) withdraw(ctx context.Context, relay *v1alpha1.DynamoPoolRel cleanupErrors = append(cleanupErrors, err) } } + // The pool's exports go with the Relay's Service, whatever the reason. + if err := r.withdrawExports(ctx, relay); err != nil { + cleanupErrors = append(cleanupErrors, err) + } if err := r.setStatus(ctx, relay, false, reason, message, "", 0, len(cleanupErrors) == 0); err != nil { cleanupErrors = append(cleanupErrors, err) } diff --git a/deploy/operator/internal/controller/poolrelay/exports.go b/deploy/operator/internal/controller/poolrelay/exports.go new file mode 100644 index 000000000000..cd9b997f4db6 --- /dev/null +++ b/deploy/operator/internal/controller/poolrelay/exports.go @@ -0,0 +1,515 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +package poolrelay + +import ( + "context" + "fmt" + "maps" + "slices" + "strings" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" + v1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" + commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/predicate" +) + +// The pool's exports: when the multiClusterExport gate is on, the Relay +// controller exports its DGD's PoolRelay and frontend Services to the other +// clusters of the ClusterSet through the configured Multi-Cluster Services (MCS) +// implementation. The Relay owns the exports, which follow its own decision: they +// are reconciled while it publishes its Service for the DGD, and withdrawn +// whenever it withdraws that Service. + +// +kubebuilder:rbac:groups=multicluster.x-k8s.io,resources=serviceexports,verbs=get;list;watch;create;patch;delete +// +kubebuilder:rbac:groups=about.k8s.io,resources=clusterproperties,verbs=get;list;watch + +// exportFieldManager is the field manager of the controller's writes to +// ServiceExports, as of its other writes. +const exportFieldManager = "dynamo-pool-relay" + +// Reasons of the Warning events that the controller records on a Relay about its +// pool's exports. Apart from a failure, they report the problems that only the +// operator can see; the MCS implementation reports on each export in the export's +// own status. +const ( + reasonInvalidServiceName = "InvalidServiceName" + reasonServiceNameTooLong = "ServiceNameTooLong" + reasonNoFrontend = "NoFrontend" + reasonClusterIDMismatch = "ClusterIDMismatch" + reasonExportFailed = "ExportFailed" +) + +// relayGroupKind is the kind that controls a pool's exports. +var relayGroupKind = v1alpha1.GroupVersion.WithKind("DynamoPoolRelay").GroupKind() + +// exportSettings configures the pool's exports, which the controller maintains +// when the multiClusterExport gate is on. +type exportSettings struct { + // cache reads unstructured objects, ServiceExports and ClusterProperties, from + // the informers that the controller's watches start. The manager's client + // reads unstructured objects from the API server. + cache client.Reader + recorder events.EventRecorder + config *configv1alpha1.OperatorConfiguration + runtimeConfig *commoncontroller.RuntimeConfig + // clusterPropertyWatched is true when the controller watches the About API's + // ClusterProperties: the MCS implementation labels imports with the cluster's + // ID in its ClusterSet, and the cluster serves the API. Otherwise the check of + // a pool's cluster ID is skipped. + clusterPropertyWatched bool +} + +// eventFilter passes the events of the namespaces that the operator manages, and +// those of the cluster-scoped ClusterProperty, whose mapper selects the Relays of +// those namespaces. +func eventFilter( + config *configv1alpha1.OperatorConfiguration, + runtimeConfig *commoncontroller.RuntimeConfig, +) predicate.Predicate { + return predicate.Or( + commoncontroller.EphemeralDeploymentEventFilter(config, runtimeConfig), + predicate.NewPredicateFuncs(func(object client.Object) bool { + return object.GetObjectKind().GroupVersionKind().GroupKind() == multicluster.ClusterPropertyGVK.GroupKind() + }), + ) +} + +// watchExports turns on the pool's exports, and adds the watches that they need: +// the ServiceExports and, with a standard implementation in a cluster that serves +// the About API, the ClusterProperty that holds the cluster's ID in its +// ClusterSet. The Relay's DGD is watched already. +func (r *reconciler) watchExports( + mgr ctrl.Manager, + controllerBuilder *builder.Builder, + config *configv1alpha1.OperatorConfiguration, + runtimeConfig *commoncontroller.RuntimeConfig, +) (*builder.Builder, error) { + r.exports = &exportSettings{ + cache: mgr.GetCache(), + recorder: mgr.GetEventRecorder("dynamo-pool-relay"), + config: config, + runtimeConfig: runtimeConfig, + } + controllerBuilder = controllerBuilder.Watches( + multicluster.NewServiceExport(runtimeConfig.Gate.ServiceExportVersion), + handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToRelays), + builder.WithPredicates(serviceExportChanged), + ) + if !multicluster.UsesClusterSetID(config.MultiCluster.MCSImplementation) { + return controllerBuilder, nil + } + served, err := clusterPropertyServed(mgr.GetRESTMapper()) + if err != nil || !served { + return controllerBuilder, err + } + r.exports.clusterPropertyWatched = true + return controllerBuilder.Watches( + multicluster.NewClusterProperty(), + handler.EnqueueRequestsFromMapFunc(r.mapClusterPropertyToRelays), + builder.WithPredicates(clusterIDPropertyEvents), + ), nil +} + +// reconcileExports exports the Services of the pool that the Relay publishes for +// dgd, and withdraws the Relay's other exports. It decides from what exists now, +// not from the events that led there, so a DGD that is deleted and created again +// under the Relay's name, quickly or not, ends up with the new DGD's exports. A +// failure is also recorded as a Warning event on the Relay. +func (r *reconciler) reconcileExports( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, +) error { + if r.exports == nil { + return nil + } + return r.exportFailed(relay, r.convergeExports(ctx, relay, dgd)) +} + +// withdrawExports withdraws every export that the Relay controls, as it withdraws +// the Relay's Service. A failure is also recorded as a Warning event on the Relay. +func (r *reconciler) withdrawExports(ctx context.Context, relay *v1alpha1.DynamoPoolRelay) error { + if r.exports == nil { + return nil + } + return r.exportFailed(relay, r.withdrawAllExports(ctx, relay)) +} + +// exportFailed records err, a failure to reconcile the pool's exports, as a +// Warning event on the Relay, and returns it. +func (r *reconciler) exportFailed(relay *v1alpha1.DynamoPoolRelay, err error) error { + if err != nil { + r.exports.recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonExportFailed, "Export", + "Failed to reconcile the pool's ServiceExports: %v", err) + } + return err +} + +// unexportedService is a pool Service that the implementation cannot export. +type unexportedService struct { + name string + problem multicluster.NameProblem + detail string +} + +func (r *reconciler) convergeExports( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, +) error { + implementation := r.exports.config.MultiCluster.MCSImplementation + existing, err := r.listExports(ctx, relay) + if err != nil { + return err + } + + // The pool exports each of its Services that the implementation can export: a + // missing export is created, and one that the Relay controls is repaired. A + // user's export, or one being deleted, stays as it is. + var exported []string + var unexported []unexportedService + for _, name := range multicluster.ServiceNames(dgd) { + if problem, detail := multicluster.CheckServiceName(implementation, name); problem != multicluster.NameExportable { + unexported = append(unexported, unexportedService{name: name, problem: problem, detail: detail}) + continue + } + current, found := existing[name] + switch { + case !found: + if err := r.createExport(ctx, relay, dgd, name); err != nil { + return err + } + case current.ControlledBy(relay.UID) && !current.Deleting: + if err := r.repairExport(ctx, relay, dgd, current); err != nil { + return err + } + } + exported = append(exported, name) + } + + // Withdraw the Relay's exports of Services that the pool no longer exports, + // such as a renamed or removed frontend's. + for _, name := range slices.Sorted(maps.Keys(existing)) { + if !slices.Contains(exported, name) { + if err := r.withdrawExport(ctx, relay, existing[name]); err != nil { + return err + } + } + } + return r.reportExportProblems(ctx, relay, dgd, unexported) +} + +func (r *reconciler) withdrawAllExports(ctx context.Context, relay *v1alpha1.DynamoPoolRelay) error { + existing, err := r.listExports(ctx, relay) + if err != nil { + return err + } + for _, name := range slices.Sorted(maps.Keys(existing)) { + if err := r.withdrawExport(ctx, relay, existing[name]); err != nil { + return err + } + } + return nil +} + +// listExports returns the ServiceExports in the Relay's namespace by name. +func (r *reconciler) listExports(ctx context.Context, relay *v1alpha1.DynamoPoolRelay) (map[string]multicluster.Export, error) { + return multicluster.ListExports(ctx, r.exports.cache, r.exports.runtimeConfig.Gate.ServiceExportVersion, relay.Namespace) +} + +// desiredExport returns the Relay's ServiceExport of the Service named name, with +// only the fields that the controller writes: the DGD label, and the Relay as its +// controller, so that the export is deleted with the Relay. +func (r *reconciler) desiredExport( + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + name string, +) (*unstructured.Unstructured, error) { + serviceExport := multicluster.RenderServiceExport( + r.exports.runtimeConfig.Gate.ServiceExportVersion, relay.Namespace, name, dgd.Name) + if err := ctrl.SetControllerReference(relay, serviceExport, r.scheme); err != nil { + return nil, fmt.Errorf("failed to set the owner of ServiceExport %s: %w", name, err) + } + return serviceExport, nil +} + +// createExport creates the Relay's ServiceExport of the Service named name. A +// create fails on any existing export, so the Relay never takes over a user's +// export of the same name that the cache has not seen yet. +func (r *reconciler) createExport( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + name string, +) error { + serviceExport, err := r.desiredExport(relay, dgd, name) + if err != nil { + return err + } + if err := r.Create(ctx, serviceExport, client.FieldOwner(exportFieldManager)); err != nil { + // The cache has not seen an export created moments ago. Its watch event + // requeues the Relay. + if apierrors.IsAlreadyExists(err) { + return nil + } + return fmt.Errorf("failed to create ServiceExport %s: %w", name, err) + } + r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "CreateServiceExport", "Create", + "Exported Service %s through MCS implementation %s", name, r.exports.config.MultiCluster.MCSImplementation) + return nil +} + +// repairExport restores the fields that the controller writes on an export the +// Relay controls, when they differ from desiredExport's, for example after +// someone removed the DGD label. Server-side apply writes only those fields, so +// the export keeps its UID, the MCS implementation's status, its finalizers, and +// other labels and annotations. +func (r *reconciler) repairExport( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + current multicluster.Export, +) error { + desired, err := r.desiredExport(relay, dgd, current.Name) + if err != nil { + return err + } + if hasDesiredFields(current, desired) { + return nil + } + + // The UID and resourceVersion tie the apply to the export whose controller was + // just checked. Once it has changed, been replaced or been deleted, the API + // server refuses the apply with a conflict instead of writing to an export + // that the Relay may no longer control, and the change's watch event requeues + // the Relay. Forcing ownership takes back a field that someone else changed: + // without it, that field would also be refused as a conflict. + desired.SetUID(current.UID) + desired.SetResourceVersion(current.ResourceVersion) + err = r.Apply(ctx, client.ApplyConfigurationFromUnstructured(desired), + client.FieldOwner(exportFieldManager), client.ForceOwnership) + if apierrors.IsConflict(err) { + return nil + } + if err != nil { + return fmt.Errorf("failed to repair ServiceExport %s: %w", current.Name, err) + } + r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "RepairServiceExport", "Apply", + "Restored the DGD label and controller reference of the export of Service %s", current.Name) + return nil +} + +// hasDesiredFields reports whether export has the labels and the controller +// reference of desired. Other labels do not matter. +func hasDesiredFields(export multicluster.Export, desired *unstructured.Unstructured) bool { + for key, value := range desired.GetLabels() { + if export.Labels[key] != value { + return false + } + } + return equality.Semantic.DeepEqual(export.Controller, metav1.GetControllerOf(desired)) +} + +// withdrawExport deletes export if the Relay controls it and it is not already +// being deleted. +func (r *reconciler) withdrawExport(ctx context.Context, relay *v1alpha1.DynamoPoolRelay, export multicluster.Export) error { + if !export.ControlledBy(relay.UID) || export.Deleting { + return nil + } + + // The UID and resourceVersion preconditions delete exactly the export just + // read, at the version whose controller was checked. A change to the export + // since, such as someone removing or transferring the controller reference, + // which keeps the UID, makes the API server refuse the delete with a conflict. + // That is retried from the change's watch event, which requeues the Relay to + // check the controller again. + serviceExport := multicluster.NewServiceExport(r.exports.runtimeConfig.Gate.ServiceExportVersion) + serviceExport.SetNamespace(relay.Namespace) + serviceExport.SetName(export.Name) + if err := r.Delete(ctx, serviceExport, client.Preconditions{ + UID: &export.UID, + ResourceVersion: &export.ResourceVersion, + }); err != nil { + if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { + return nil + } + return fmt.Errorf("failed to delete ServiceExport %s: %w", export.Name, err) + } + r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", + "Withdrew the export of Service %s", export.Name) + return nil +} + +// reportExportProblems records a Warning event on the Relay for each problem of +// the pool that only the operator can see: a Service that the implementation +// cannot export, a DGD without a frontend, and a cluster ID that differs from the +// cluster's ID in its ClusterSet. A failed read of that ID is an error. +func (r *reconciler) reportExportProblems( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + unexported []unexportedService, +) error { + recorder := r.exports.recorder + if names, detail := unexportedNames(unexported, multicluster.NameInvalid); len(names) > 0 { + recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonInvalidServiceName, "Export", + "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail) + } + if names, detail := unexportedNames(unexported, multicluster.NameTooLong); len(names) > 0 { + recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonServiceNameTooLong, "Export", + "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", ")) + } + if len(multicluster.FrontendServiceNames(dgd)) == 0 { + recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonNoFrontend, "Export", + "DGD %s has no frontend component, so the Global Router has nowhere to send the pool's requests", dgd.Name) + } + + // A standard implementation labels the pool's imports with the cluster's ID in + // its ClusterSet, and the Global Discovery Service joins the pool's + // registration to them only under the same cluster ID. + if !r.exports.clusterPropertyWatched { + return nil + } + clusterID, err := multicluster.ReadClusterID(ctx, r.exports.cache) + if err != nil { + return err + } + if clusterID != "" && clusterID != relay.Spec.ClusterID { + recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonClusterIDMismatch, "Export", + "clusterId %q differs from the ID %q that ClusterProperty %s gives this cluster in its ClusterSet; "+ + "the Global Discovery Service lists the pool only when they match", + relay.Spec.ClusterID, clusterID, multicluster.ClusterIDProperty) + } + return nil +} + +// unexportedNames returns the names of the services with problem, and the first +// one's detail. +func unexportedNames(services []unexportedService, problem multicluster.NameProblem) ([]string, string) { + var names []string + detail := "" + for _, service := range services { + if service.problem != problem { + continue + } + if len(names) == 0 { + detail = service.detail + } + names = append(names, service.name) + } + return names, detail +} + +// mapServiceExportToRelays requeues the Relay that controls the export, and each +// Relay in the export's namespace whose DGD exports a Service of the export's +// name: such a Relay takes over when a user's export of that Service is deleted. +func (r *reconciler) mapServiceExportToRelays(ctx context.Context, export client.Object) []ctrl.Request { + var requests []ctrl.Request + add := func(name string) { + request := ctrl.Request{Namespace: export.GetNamespace(), Name: name} + if !slices.Contains(requests, request) { + requests = append(requests, request) + } + } + if controller := metav1.GetControllerOf(export); controller != nil && + schema.FromAPIVersionAndKind(controller.APIVersion, controller.Kind).GroupKind() == relayGroupKind { + add(controller.Name) + } + + relays := &v1alpha1.DynamoPoolRelayList{} + if err := r.List(ctx, relays, client.InNamespace(export.GetNamespace())); err != nil { + log.FromContext(ctx).Error(err, "Failed to list DynamoPoolRelays for ServiceExport event") + return requests + } + for i := range relays.Items { + dgd := &v1beta1.DynamoGraphDeployment{} + if err := r.Get(ctx, client.ObjectKeyFromObject(&relays.Items[i]), dgd); err != nil { + continue + } + if slices.Contains(multicluster.ServiceNames(dgd), export.GetName()) { + add(relays.Items[i].Name) + } + } + return requests +} + +// mapClusterPropertyToRelays requeues every Relay in a namespace that the +// operator manages: each pool's cluster ID is compared with the cluster's ID in +// its ClusterSet. +func (r *reconciler) mapClusterPropertyToRelays(ctx context.Context, _ client.Object) []ctrl.Request { + relays := &v1alpha1.DynamoPoolRelayList{} + if err := r.List(ctx, relays); err != nil { + log.FromContext(ctx).Error(err, "Failed to list DynamoPoolRelays for ClusterProperty event") + return nil + } + requests := make([]ctrl.Request, 0, len(relays.Items)) + for i := range relays.Items { + relay := &relays.Items[i] + if commoncontroller.NamespaceAllowed(r.exports.config, r.exports.runtimeConfig, relay, relay.Namespace) { + requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)}) + } + } + return requests +} + +// serviceExportChanged passes any update of a ServiceExport with a controller, +// before or after the update: the Relay that controls the export repairs it, and +// one that stopped controlling it leaves it alone. This also covers a change that +// made a repair or a withdrawal conflict. The creation and deletion of any export +// pass too. +var serviceExportChanged = predicate.Funcs{UpdateFunc: func(update event.UpdateEvent) bool { + return metav1.GetControllerOf(update.ObjectOld) != nil || metav1.GetControllerOf(update.ObjectNew) != nil +}} + +// clusterIDPropertyEvents passes the events of the ClusterProperty that holds the +// cluster's ID in its ClusterSet: its creation, its deletion and a change of its +// value. +var clusterIDPropertyEvents = predicate.Funcs{ + CreateFunc: func(e event.CreateEvent) bool { return e.Object.GetName() == multicluster.ClusterIDProperty }, + DeleteFunc: func(e event.DeleteEvent) bool { return e.Object.GetName() == multicluster.ClusterIDProperty }, + UpdateFunc: clusterIDPropertyChanged, + GenericFunc: func(event.GenericEvent) bool { return false }, +} + +// clusterIDPropertyChanged passes the updates of the ClusterProperty that holds +// the cluster's ID in its ClusterSet that change its value. +func clusterIDPropertyChanged(update event.UpdateEvent) bool { + oldProperty, oldOK := update.ObjectOld.(*unstructured.Unstructured) + newProperty, newOK := update.ObjectNew.(*unstructured.Unstructured) + return oldOK && newOK && newProperty.GetName() == multicluster.ClusterIDProperty && + multicluster.ClusterPropertyValue(oldProperty) != multicluster.ClusterPropertyValue(newProperty) +} + +// clusterPropertyServed reports whether the cluster serves the About API's +// ClusterProperty, which the controller then watches. +func clusterPropertyServed(mapper meta.RESTMapper) (bool, error) { + _, err := mapper.RESTMapping(multicluster.ClusterPropertyGVK.GroupKind(), multicluster.ClusterPropertyGVK.Version) + if meta.IsNoMatchError(err) { + return false, nil + } + if err != nil { + return false, fmt.Errorf("discover the ClusterProperty API for the pool's exports: %w", err) + } + return true, nil +} diff --git a/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go new file mode 100644 index 000000000000..c3fa301c20b0 --- /dev/null +++ b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go @@ -0,0 +1,446 @@ +//go:build !clustertest + +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +package poolrelay + +import ( + "context" + "fmt" + "maps" + "os" + "path/filepath" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" + v1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" + commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + corev1 "k8s.io/api/core/v1" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/rest" + "k8s.io/utils/ptr" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + ctrlconfig "sigs.k8s.io/controller-runtime/pkg/config" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/metrics/server" + "sigs.k8s.io/yaml" +) + +// testCRD returns a CustomResourceDefinition that serves kind in group at +// v1alpha1 and keeps any field: a stand-in for the MCS and About APIs, whose +// CRDs the operator does not ship. +func testCRD(group, kind, plural string, scope apiextensionsv1.ResourceScope) *apiextensionsv1.CustomResourceDefinition { + return &apiextensionsv1.CustomResourceDefinition{ + Name: plural + "." + group, + Spec: apiextensionsv1.CustomResourceDefinitionSpec{ + Group: group, + Names: apiextensionsv1.CustomResourceDefinitionNames{ + Kind: kind, ListKind: kind + "List", Plural: plural, Singular: strings.ToLower(kind), + }, + Scope: scope, + Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{ + Name: "v1alpha1", Served: true, Storage: true, + Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ + Type: "object", XPreserveUnknownFields: new(true), + }}, + Subresources: &apiextensionsv1.CustomResourceSubresources{ + Status: &apiextensionsv1.CustomResourceSubresourceStatus{}, + }, + }}, + }, + } +} + +// startExportAPIServer starts an API server with the operator's CRDs and +// stand-ins for the ServiceExport and ClusterProperty APIs. It runs no garbage +// collector. +func startExportAPIServer(t *testing.T) *rest.Config { + t.Helper() + // about.k8s.io is a protected API group, so its CRD needs the approval + // annotation that the About API's own CRD has. + clusterProperties := testCRD(multicluster.ClusterPropertyGVK.Group, multicluster.ClusterPropertyGVK.Kind, + "clusterproperties", apiextensionsv1.ClusterScoped) + clusterProperties.Annotations = map[string]string{ + apiextensionsv1.KubeAPIApprovedAnnotation: "https://github.com/kubernetes/enhancements/pull/3084", + } + environment := &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("..", "..", "..", "config", "crd", "bases")}, + CRDs: []*apiextensionsv1.CustomResourceDefinition{ + testCRD(multicluster.ServiceExportGroupKind.Group, multicluster.ServiceExportGroupKind.Kind, + "serviceexports", apiextensionsv1.NamespaceScoped), + clusterProperties, + }, + ErrorIfCRDPathMissing: true, + } + config, err := environment.Start() + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + if err := environment.Stop(); err != nil { + t.Error(err) + } + }) + return config +} + +// startRelayController starts a manager that runs the Relay controller with the +// pool's exports on, for standard MCS, and returns a function that stops the +// manager and waits for it. wrap, when set, wraps the manager's client. +func startRelayController( + t *testing.T, + config *rest.Config, + wrap func(client.Client) client.Client, +) func() { + t.Helper() + manager, err := ctrl.NewManager(config, ctrl.Options{ + Scheme: newExportTestScheme(t), + Metrics: server.Options{BindAddress: "0"}, + // A test can start managers one after another, with the same controllers. + Controller: ctrlconfig.Controller{SkipNameValidation: new(true)}, + NewClient: func(restConfig *rest.Config, options client.Options) (client.Client, error) { + c, err := client.New(restConfig, options) + if err != nil || wrap == nil { + return c, err + } + return wrap(c), nil + }, + }) + if err != nil { + t.Fatal(err) + } + if err := Setup(manager, &configv1alpha1.OperatorConfiguration{ + Discovery: configv1alpha1.DiscoveryConfiguration{Backend: configv1alpha1.DiscoveryBackendKubernetes}, + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: configv1alpha1.MCSImplementationStandard}, + }, &commoncontroller.RuntimeConfig{ + Gate: features.Gates{MultiClusterExport: true, ServiceExportVersion: exportTestVersion}, + }); err != nil { + t.Fatal(err) + } + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- manager.Start(ctx) }() + var once sync.Once + stop := func() { + once.Do(func() { + cancel() + if err := <-done; err != nil { + t.Error(err) + } + }) + } + t.Cleanup(stop) + return stop +} + +// eventually polls check until it reports done, and fails the test with what +// after 30 seconds. +func eventually(t *testing.T, what string, check func() (bool, error)) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for { + done, err := check() + if err != nil { + t.Fatalf("%s: %v", what, err) + } + if done { + return + } + if time.Now().After(deadline) { + t.Fatalf("timed out waiting for %s", what) + } + time.Sleep(100 * time.Millisecond) + } +} + +// readSampleRelay returns the checked-in DynamoPoolRelay sample. +func readSampleRelay(t *testing.T) *v1alpha1.DynamoPoolRelay { + t.Helper() + sample, err := os.ReadFile(filepath.Join("..", "..", "..", "samples", "dynamo-pool-relay.yaml")) + if err != nil { + t.Fatal(err) + } + relay := &v1alpha1.DynamoPoolRelay{} + if err := yaml.UnmarshalStrict(sample, relay); err != nil { + t.Fatal(err) + } + return relay +} + +// createNamespace creates the namespace called name. +func createNamespace(t *testing.T, c client.Client, name string) { + t.Helper() + namespace := &corev1.Namespace{} + namespace.Name = name + if err := c.Create(context.Background(), namespace); err != nil { + t.Fatal(err) + } +} + +// createDGD creates the DGD of the Relay's name with a worker and a frontend +// component named frontend, and returns it. +func createDGD(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, frontend string) *v1beta1.DynamoGraphDeployment { + t.Helper() + dgd := &v1beta1.DynamoGraphDeployment{ + Name: relay.Name, Namespace: relay.Namespace, + Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + {ComponentName: "decode", ComponentType: v1beta1.ComponentTypeWorker}, + {ComponentName: frontend, ComponentType: v1beta1.ComponentTypeFrontend}, + }}, + } + if err := c.Create(context.Background(), dgd); err != nil { + t.Fatal(err) + } + return dgd +} + +// reportTerminalFailure writes the status that the DGD controller writes when a +// DGD's workloads cannot be reconciled at all, with message. +func reportTerminalFailure(t *testing.T, c client.Client, dgd *v1beta1.DynamoGraphDeployment, message string) { + t.Helper() + if err := c.Get(context.Background(), client.ObjectKeyFromObject(dgd), dgd); err != nil { + t.Fatal(err) + } + dgd.Status.State = v1beta1.DGDStateFailed + meta.SetStatusCondition(&dgd.Status.Conditions, metav1.Condition{ + Type: "Ready", Status: metav1.ConditionFalse, Reason: "UnsupportedWorkloadProvider", Message: message, + }) + if err := c.Status().Update(context.Background(), dgd); err != nil { + t.Fatal(err) + } +} + +// ownedByRelay reports whether relay controls export and blocks its own deletion +// on it, so that the garbage collector deletes the export with the Relay. +func ownedByRelay(export *unstructured.Unstructured, relay *v1alpha1.DynamoPoolRelay) bool { + controller := metav1.GetControllerOf(export) + return controller != nil && controller.UID == relay.UID && ptr.Deref(controller.BlockOwnerDeletion, false) +} + +// exportsByName returns the ServiceExports in namespace by name. +func exportsByName(t *testing.T, c client.Client, namespace string) map[string]*unstructured.Unstructured { + t.Helper() + exports := map[string]*unstructured.Unstructured{} + items := listServiceExports(t, c, exportTestVersion, namespace) + for i := range items { + exports[items[i].GetName()] = &items[i] + } + return exports +} + +// waitForExports waits until the ServiceExports in the Relay's namespace are +// exactly names, none being deleted, and each owned by relay. +func waitForExports(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, names ...string) { + t.Helper() + eventually(t, "the exports "+strings.Join(names, ", "), func() (bool, error) { + exports := exportsByName(t, c, relay.Namespace) + if len(exports) != len(names) { + return false, nil + } + for name, export := range exports { + if !slices.Contains(names, name) || export.GetDeletionTimestamp() != nil || !ownedByRelay(export, relay) { + return false, nil + } + } + return true, nil + }) +} + +// waitForRelayService waits until the controller publishes the Relay's Service, +// or withdraws it. +func waitForRelayService(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, published bool) { + t.Helper() + key := types.NamespacedName{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + eventually(t, fmt.Sprintf("the Relay's Service to be published: %v", published), func() (bool, error) { + err := c.Get(context.Background(), key, &corev1.Service{}) + if apierrors.IsNotFound(err) { + return !published, nil + } + return published, err + }) +} + +// waitForDeletion waits until obj is gone. +func waitForDeletion(t *testing.T, c client.Client, obj client.Object) { + t.Helper() + eventually(t, obj.GetName()+" to be deleted", func() (bool, error) { + err := c.Get(context.Background(), client.ObjectKeyFromObject(obj), obj) + if apierrors.IsNotFound(err) { + return true, nil + } + return false, err + }) +} + +//nolint:gocyclo // This test follows one Relay through the replacement of its DGD against the API server. +func TestRelayControllerFollowsAReplacedDGD(t *testing.T) { + if os.Getenv("KUBEBUILDER_ASSETS") == "" { + t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") + } + ctx := context.Background() + config := startExportAPIServer(t) + apiClient, err := client.New(config, client.Options{Scheme: newExportTestScheme(t)}) + if err != nil { + t.Fatal(err) + } + stop := startRelayController(t, config, nil) + + t.Log("Create a DGD with a frontend component, and its DynamoPoolRelay from the checked-in sample") + relay := readSampleRelay(t) + createNamespace(t, apiClient, relay.Namespace) + first := createDGD(t, apiClient, relay, "Frontend") + if err := apiClient.Create(ctx, relay); err != nil { + t.Fatal(err) + } + relayUID := relay.UID + poolRelayExport := serviceName(relay.Name) + waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-frontend") + waitForRelayService(t, apiClient, relay, true) + + t.Log("An MCS implementation holds the PoolRelay export with a finalizer, so its deletion will stay pending") + held := exportsByName(t, apiClient, relay.Namespace)[poolRelayExport] + heldUID := held.GetUID() + held.SetFinalizers([]string{"example.com/mcs-hold"}) + if err := apiClient.Update(ctx, held); err != nil { + t.Fatal(err) + } + + t.Log("Delete the DGD: the Relay's Service and frontend export are withdrawn, and the PoolRelay export's deletion is pending") + if err := apiClient.Delete(ctx, first); err != nil { + t.Fatal(err) + } + eventually(t, "the exports to be withdrawn", func() (bool, error) { + exports := exportsByName(t, apiClient, relay.Namespace) + pending, found := exports[poolRelayExport] + return len(exports) == 1 && found && pending.GetDeletionTimestamp() != nil, nil + }) + waitForRelayService(t, apiClient, relay, false) + + t.Log("Create a new DGD of the Relay's name with another frontend: its export is created while the old PoolRelay export is pending") + second := createDGD(t, apiClient, relay, "Web") + if second.UID == first.UID { + t.Fatal("the new DGD has the old DGD's UID") + } + eventually(t, "the new frontend's export next to the pending one", func() (bool, error) { + exports := exportsByName(t, apiClient, relay.Namespace) + web, webFound := exports[relay.Name+"-web"] + pending, pendingFound := exports[poolRelayExport] + return len(exports) == 2 && webFound && ownedByRelay(web, relay) && + pendingFound && pending.GetUID() == heldUID && pending.GetDeletionTimestamp() != nil, nil + }) + waitForRelayService(t, apiClient, relay, true) + + t.Log("The MCS implementation lets go of the old export: a new PoolRelay export is created with no other change") + held = exportsByName(t, apiClient, relay.Namespace)[poolRelayExport] + held.SetFinalizers(nil) + if err := apiClient.Update(ctx, held); err != nil { + t.Fatal(err) + } + waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-web") + if uid := exportsByName(t, apiClient, relay.Namespace)[poolRelayExport].GetUID(); uid == heldUID { + t.Fatal("the PoolRelay export was not created again") + } + if err := apiClient.Get(ctx, client.ObjectKeyFromObject(relay), relay); err != nil { + t.Fatal(err) + } + if relay.UID != relayUID { + t.Fatal("the DynamoPoolRelay was created again") + } + + t.Log("Status updates of the DGD, such as a terminal workload failure, write nothing to the exports") + versions := func() map[string]string { + byName := map[string]string{} + for name, export := range exportsByName(t, apiClient, relay.Namespace) { + byName[name] = export.GetResourceVersion() + } + return byName + } + before := versions() + for attempt := range 3 { + reportTerminalFailure(t, apiClient, second, fmt.Sprintf("Grove is disabled (report %d)", attempt)) + } + time.Sleep(2 * time.Second) + if after := versions(); !maps.Equal(before, after) { + t.Fatalf("DGD status updates changed the exports: %v -> %v", before, after) + } + + t.Log("With the controller stopped, replace the DGD at once, so that no cache sees it missing") + stop() + if err := apiClient.Delete(ctx, second); err != nil { + t.Fatal(err) + } + waitForDeletion(t, apiClient, second) + createDGD(t, apiClient, relay, "Api") + startRelayController(t, config, nil) + waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-api") + waitForRelayService(t, apiClient, relay, true) +} + +// failingCreates refuses the first ServiceExport creates, as an API server that +// is briefly unavailable does, and counts the attempts. +type failingCreates struct { + client.Client + remaining atomic.Int32 + attempts atomic.Int32 +} + +func (c *failingCreates) Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error { + if obj.GetObjectKind().GroupVersionKind().GroupKind() == multicluster.ServiceExportGroupKind { + c.attempts.Add(1) + if c.remaining.Add(-1) >= 0 { + return apierrors.NewServiceUnavailable("injected failure") + } + } + return c.Client.Create(ctx, obj, opts...) +} + +func TestRelayControllerRetriesFailedExports(t *testing.T) { + if os.Getenv("KUBEBUILDER_ASSETS") == "" { + t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") + } + ctx := context.Background() + config := startExportAPIServer(t) + apiClient, err := client.New(config, client.Options{Scheme: newExportTestScheme(t)}) + if err != nil { + t.Fatal(err) + } + failing := &failingCreates{} + failing.remaining.Store(2) + startRelayController(t, config, func(c client.Client) client.Client { + failing.Client = c + return failing + }) + + t.Log("Create a pool whose DGD reports a terminal workload failure, while the API server refuses the first exports") + relay := readSampleRelay(t) + createNamespace(t, apiClient, relay.Namespace) + dgd := createDGD(t, apiClient, relay, "Frontend") + reportTerminalFailure(t, apiClient, dgd, "Grove is disabled") + if err := apiClient.Create(ctx, relay); err != nil { + t.Fatal(err) + } + + t.Log("The controller retries on its own, and the exports converge with no other change") + waitForExports(t, apiClient, relay, serviceName(relay.Name), relay.Name+"-frontend") + if attempts := failing.attempts.Load(); attempts <= 2 { + t.Fatalf("the exports converged after %d create attempts, without retrying the 2 refused ones", attempts) + } +} diff --git a/deploy/operator/internal/controller/poolrelay/exports_test.go b/deploy/operator/internal/controller/poolrelay/exports_test.go new file mode 100644 index 000000000000..08f057d12835 --- /dev/null +++ b/deploy/operator/internal/controller/poolrelay/exports_test.go @@ -0,0 +1,1090 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + */ + +package poolrelay + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" + v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" + v1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" + commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/uuid" + "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/event" +) + +// exportTestVersion is the multicluster.x-k8s.io version that the tests' clusters +// serve. +const exportTestVersion = "v1alpha1" + +// newExportTestScheme returns a scheme with the kinds that the controller reads +// and writes. +func newExportTestScheme(t *testing.T) *runtime.Scheme { + t.Helper() + scheme := runtime.NewScheme() + for _, add := range []func(*runtime.Scheme) error{ + v1alpha1.AddToScheme, v1beta1.AddToScheme, appsv1.AddToScheme, corev1.AddToScheme, rbacv1.AddToScheme, + } { + if err := add(scheme); err != nil { + t.Fatalf("add to scheme: %v", err) + } + } + return scheme +} + +// newExportTestClient returns a fake client that serves the scheme's kinds, the +// ServiceExport kind at version only, and the ClusterProperty kind, which the +// scheme does not register. +func newExportTestClient( + scheme *runtime.Scheme, + version string, + funcs interceptor.Funcs, + objects ...client.Object, +) client.WithWatch { + mapper := meta.NewDefaultRESTMapper(nil) + for gvk := range scheme.AllKnownTypes() { + mapper.Add(gvk, meta.RESTScopeNamespace) + } + mapper.Add(multicluster.ServiceExportGroupKind.WithVersion(version), meta.RESTScopeNamespace) + mapper.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) + return fake.NewClientBuilder(). + WithScheme(scheme). + WithRESTMapper(mapper). + WithStatusSubresource(&v1alpha1.DynamoPoolRelay{}, &v1beta1.DynamoGraphDeployment{}, &appsv1.Deployment{}). + WithInterceptorFuncs(funcs). + WithObjects(objects...). + Build() +} + +// newExportTestDGD returns a DGD named name in namespace inference that a Relay +// can publish: a decode worker, and a frontend component for each of frontends. +func newExportTestDGD(name string, frontends ...string) *v1beta1.DynamoGraphDeployment { + dgd := &v1beta1.DynamoGraphDeployment{ + Name: name, Namespace: "inference", UID: uuid.NewUUID(), + Spec: v1beta1.DynamoGraphDeploymentSpec{Components: []v1beta1.DynamoComponentDeploymentSharedSpec{ + {ComponentName: "decode", ComponentType: v1beta1.ComponentTypeWorker}, + }}, + Status: v1beta1.DynamoGraphDeploymentStatus{Components: map[string]v1beta1.ComponentReplicaStatus{ + "decode": {RuntimeNamespace: "inference-" + name + "-hash"}, + }}, + } + for _, frontend := range frontends { + dgd.Spec.Components = append(dgd.Spec.Components, v1beta1.DynamoComponentDeploymentSharedSpec{ + ComponentName: frontend, ComponentType: v1beta1.ComponentTypeFrontend, + }) + } + return dgd +} + +// newExportTestRelay returns the Relay that makes dgd a pool of cluster east. +func newExportTestRelay(dgd *v1beta1.DynamoGraphDeployment) *v1alpha1.DynamoPoolRelay { + return &v1alpha1.DynamoPoolRelay{ + Name: dgd.Name, Namespace: dgd.Namespace, UID: uuid.NewUUID(), + Spec: relaySpec("east"), + } +} + +// ownedExport returns the export at version of the Service named name that relay +// controls, labeled with the DGD of the Relay's name. +func ownedExport( + t *testing.T, + scheme *runtime.Scheme, + version string, + relay *v1alpha1.DynamoPoolRelay, + name string, +) *unstructured.Unstructured { + t.Helper() + export := multicluster.RenderServiceExport(version, relay.Namespace, name, relay.Name) + if err := controllerutil.SetControllerReference(relay, export, scheme); err != nil { + t.Fatalf("set the owner of ServiceExport %s: %v", name, err) + } + return export +} + +// listServiceExports returns the ServiceExports in namespace, read at version. +func listServiceExports(t *testing.T, c client.Client, version, namespace string) []unstructured.Unstructured { + t.Helper() + list := &unstructured.UnstructuredList{} + list.SetGroupVersionKind(multicluster.ServiceExportGroupKind.WithVersion(version).GroupVersion().WithKind("ServiceExportList")) + if err := c.List(context.Background(), list, client.InNamespace(namespace)); err != nil { + t.Fatalf("list ServiceExports: %v", err) + } + return list.Items +} + +// exportNames returns the names of exports. +func exportNames(exports []unstructured.Unstructured) []string { + var names []string + for i := range exports { + names = append(names, exports[i].GetName()) + } + return names +} + +// newExportTestReconciler returns the Relay controller with the pool's exports on +// for implementation, reading and writing exports at version through c. +func newExportTestReconciler( + c client.WithWatch, + recorder events.EventRecorder, + implementation configv1alpha1.MCSImplementation, + version string, +) *reconciler { + return &reconciler{ + Client: c, + scheme: c.Scheme(), + defaultBackend: configv1alpha1.DiscoveryBackendKubernetes, + exports: &exportSettings{ + cache: c, + recorder: recorder, + config: &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: implementation}, + }, + runtimeConfig: &commoncontroller.RuntimeConfig{ + Gate: features.Gates{MultiClusterExport: true, ServiceExportVersion: version}, + }, + }, + } +} + +// drainEvents returns the events recorder holds. +func drainEvents(recorder *events.FakeRecorder) []string { + var recorded []string + for len(recorder.Events) > 0 { + recorded = append(recorded, <-recorder.Events) + } + return recorded +} + +// eventReasons returns the reason of each recorded event, in order. +func eventReasons(recorded []string) []string { + var reasons []string + for _, recordedEvent := range recorded { + if fields := strings.Fields(recordedEvent); len(fields) >= 2 { + reasons = append(reasons, fields[1]) + } + } + return reasons +} + +// countReasons returns how many of the recorded events have reason. +func countReasons(recorded []string, reason string) int { + count := 0 + for _, recordedReason := range eventReasons(recorded) { + if recordedReason == reason { + count++ + } + } + return count +} + +func TestReconcileExports(t *testing.T) { + // -pool-relay is 56 characters, one more than Karmada imports; + // -frontend is 54. + longName := strings.Repeat("a", 45) + + for _, tc := range []struct { + name string + dgdName string + frontends []string + implementation configv1alpha1.MCSImplementation + ownedBefore []string + foreign []string + // ownedByAnother are exports that another Relay controls. + ownedByAnother []string + wantOwned []string + // wantReasons are the reasons of the events of the first reconcile. The + // second records only its warnings again. + wantReasons []string + }{ + { + name: "a pool exports its PoolRelay and frontend Services", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"green-pool-relay", "green-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + }, + { + name: "each frontend component's Service is exported under its own name", + dgdName: "green", frontends: []string{"Frontend", "Web.v2"}, + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{"green-pool-relay", "green-frontend", "green-web-v2"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport", "CreateServiceExport"}, + }, + { + name: "a renamed frontend's export moves to its new Service", + dgdName: "green", frontends: []string{"Web"}, + implementation: configv1alpha1.MCSImplementationKarmada, + ownedBefore: []string{"green-pool-relay", "green-frontend"}, + wantOwned: []string{"green-pool-relay", "green-web"}, + wantReasons: []string{"CreateServiceExport", "DeleteServiceExport"}, + }, + { + name: "a pool without a frontend exports its PoolRelay Service and warns", + dgdName: "green", + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport", reasonNoFrontend}, + }, + { + name: "a DGD with a dot in its name has no PoolRelay Service to export", + dgdName: "green3-0.6b", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{"green3-0-6b-frontend"}, + wantReasons: []string{"CreateServiceExport", reasonInvalidServiceName}, + }, + { + name: "an export a user created is neither adopted nor deleted", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + foreign: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport"}, + }, + { + name: "another Relay's export of a Service the pool no longer has is left alone", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + ownedByAnother: []string{"green-old"}, + wantOwned: []string{"green-pool-relay", "green-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + }, + { + name: "Karmada skips a Service name it cannot import", + dgdName: longName, frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + wantOwned: []string{longName + "-frontend"}, + wantReasons: []string{"CreateServiceExport", reasonServiceNameTooLong}, + }, + { + name: "standard MCS exports the same long names", + dgdName: longName, frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create the Relay, its DGD, and the exports that exist before reconciling") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD(tc.dgdName, tc.frontends...) + relay := newExportTestRelay(dgd) + objects := []client.Object{relay, dgd} + for _, name := range tc.ownedBefore { + objects = append(objects, ownedExport(t, scheme, exportTestVersion, relay, name)) + } + for _, name := range tc.foreign { + objects = append(objects, multicluster.RenderServiceExport(exportTestVersion, dgd.Namespace, name, dgd.Name)) + } + another := newExportTestRelay(newExportTestDGD("another")) + for _, name := range tc.ownedByAnother { + objects = append(objects, ownedExport(t, scheme, exportTestVersion, another, name)) + } + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, objects...) + + t.Log("Reconcile the pool's exports, as the Relay controller does when it publishes the Relay") + recorder := events.NewFakeRecorder(20) + r := newExportTestReconciler(fakeClient, recorder, tc.implementation, exportTestVersion) + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + first := eventReasons(drainEvents(recorder)) + g.Expect(first).To(gomega.ConsistOf(tc.wantReasons)) + + t.Log("Reconcile again: every export is in place, so only the warnings are recorded again") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + var warnings []string + for _, reason := range first { + if !strings.HasSuffix(reason, "ServiceExport") { + warnings = append(warnings, reason) + } + } + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf(warnings)) + + t.Log("Check which exports the Relay controls, and that other exports are untouched") + var owned, others []string + for i, export := range listServiceExports(t, fakeClient, exportTestVersion, dgd.Namespace) { + if !metav1.IsControlledBy(&export, relay) { + others = append(others, export.GetName()) + continue + } + owned = append(owned, export.GetName()) + g.Expect(export.GetLabels()).To(gomega.HaveKeyWithValue(commonconsts.KubeLabelDynamoGraphDeploymentName, dgd.Name), i) + } + g.Expect(owned).To(gomega.ConsistOf(tc.wantOwned)) + g.Expect(others).To(gomega.ConsistOf(append(append([]string{}, tc.foreign...), tc.ownedByAnother...))) + }) + } +} + +func TestWithdrawExports(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a Relay's exports, one of them already being deleted, a user's export and another Relay's") + scheme := newExportTestScheme(t) + relay := newExportTestRelay(newExportTestDGD("green")) + terminating := ownedExport(t, scheme, exportTestVersion, relay, "green-old") + terminating.SetFinalizers([]string{"example.com/hold"}) + terminating.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) + another := newExportTestRelay(newExportTestDGD("blue")) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend"), + terminating, + multicluster.RenderServiceExport(exportTestVersion, relay.Namespace, "green-web", relay.Name), + ownedExport(t, scheme, exportTestVersion, another, "blue-frontend")) + + t.Log("Withdraw the Relay's exports, as the Relay controller does when it withdraws the Relay") + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + g.Expect(r.withdrawExports(ctx, relay)).To(gomega.Succeed()) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("DeleteServiceExport", "DeleteServiceExport")) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-old", "green-web", "blue-frontend")) +} + +func TestRelayControllerMaintainsThePoolsExports(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a DGD that a Relay can publish, and its Relay") + scheme := newExportTestScheme(t) + first := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(first) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, first) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(50), configv1alpha1.MCSImplementationKarmada, exportTestVersion) + request := ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)} + serviceKey := client.ObjectKey{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + reconcileRelay := func() { + _, err := r.Reconcile(ctx, request) + g.Expect(err).NotTo(gomega.HaveOccurred()) + } + exports := func() []string { + return exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace)) + } + + t.Log("Reconcile: the Relay's Service is published, and the pool's exports are created") + reconcileRelay() + g.Expect(fakeClient.Get(ctx, serviceKey, &corev1.Service{})).To(gomega.Succeed()) + g.Expect(exports()).To(gomega.ConsistOf("green-pool-relay", "green-frontend")) + + t.Log("Delete the DGD: the Relay's Service and the pool's exports are withdrawn, and the Relay stays") + g.Expect(fakeClient.Delete(ctx, first)).To(gomega.Succeed()) + reconcileRelay() + g.Expect(apierrors.IsNotFound(fakeClient.Get(ctx, serviceKey, &corev1.Service{}))).To(gomega.BeTrue()) + g.Expect(exports()).To(gomega.BeEmpty()) + + t.Log("Create a new DGD of the Relay's name with another frontend: the Relay publishes it, and its exports") + second := newExportTestDGD("green", "Web") + g.Expect(fakeClient.Create(ctx, second)).To(gomega.Succeed()) + reconcileRelay() + g.Expect(fakeClient.Get(ctx, serviceKey, &corev1.Service{})).To(gomega.Succeed()) + g.Expect(exports()).To(gomega.ConsistOf("green-pool-relay", "green-web")) + + t.Log("Leave the DGD without workers: the Relay withdraws, and the pool's exports with it") + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(second), second)).To(gomega.Succeed()) + second.Spec.Components = second.Spec.Components[1:] + g.Expect(fakeClient.Update(ctx, second)).To(gomega.Succeed()) + reconcileRelay() + g.Expect(apierrors.IsNotFound(fakeClient.Get(ctx, serviceKey, &corev1.Service{}))).To(gomega.BeTrue()) + g.Expect(exports()).To(gomega.BeEmpty()) +} + +func TestPoolExportsDoNotWaitForTheRelaysDeployment(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a publishable Relay while the API server refuses its Deployment") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + refused := apierrors.NewServiceUnavailable("the API server is restarting") + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if _, ok := obj.(*appsv1.Deployment); ok { + return refused + } + return c.Create(ctx, obj, opts...) + }, + }, relay, dgd) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(20), configv1alpha1.MCSImplementationKarmada, exportTestVersion) + + t.Log("Reconcile: the Deployment's failure is returned, and the pool's exports are created anyway") + _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)}) + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-pool-relay", "green-frontend")) +} + +func TestOwnershipConflictWithdrawsThePoolsExports(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a published pool's exports, and a user's Service that takes the Relay Service's name") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + foreign := &corev1.Service{Name: serviceName(relay.Name), Namespace: relay.Namespace} + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, dgd, foreign, + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend")) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(20), configv1alpha1.MCSImplementationKarmada, exportTestVersion) + + t.Log("Reconcile: the conflict withdraws the Relay, and the pool's exports with it") + _, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)}) + g.Expect(errors.Is(err, errOwnershipConflict)).To(gomega.BeTrue(), "%v", err) + g.Expect(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace)).To(gomega.BeEmpty()) +} + +func TestReconcileExportsUsesTheResolvedVersion(t *testing.T) { + g := gomega.NewGomegaWithT(t) + const version = "v1beta1" + + t.Log("Serve ServiceExports at a version other than the tests' default, with an export to withdraw") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Web") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, version, interceptor.Funcs{}, + dgd, relay, ownedExport(t, scheme, version, relay, "green-frontend")) + + t.Log("Reconcile with the gates resolved to that version: exports are listed, created and withdrawn at it") + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, version) + g.Expect(r.reconcileExports(context.Background(), relay, dgd)).To(gomega.Succeed()) + g.Expect(recorder.Events).To(gomega.HaveLen(3)) + exports := listServiceExports(t, fakeClient, version, dgd.Namespace) + for i := range exports { + g.Expect(exports[i].GetAPIVersion()).To(gomega.Equal("multicluster.x-k8s.io/" + version)) + } + g.Expect(exportNames(exports)).To(gomega.ConsistOf("green-pool-relay", "green-web")) +} + +func TestReconcileExportsToleratesAStaleCache(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("The cache has not yet seen exports that the API server already has") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + return apierrors.NewAlreadyExists( + schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, obj.GetName()) + }, + }, dgd, relay) + + t.Log("Reconcile: AlreadyExists is no error, and nothing is announced") + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) + g.Expect(r.reconcileExports(context.Background(), relay, dgd)).To(gomega.Succeed()) + g.Expect(recorder.Events).To(gomega.BeEmpty()) +} + +func TestPoolExportsRetryFailedWrites(t *testing.T) { + unavailable := apierrors.NewServiceUnavailable("the API server is restarting") + for _, tc := range []struct { + name string + // failOnce fails the first write of the case's kind. + failOnce func(failed *bool) interceptor.Funcs + // withdraw withdraws the exports instead of reconciling them. + withdraw bool + want []string + }{ + { + name: "a failed create", + failOnce: func(failed *bool) interceptor.Funcs { + return interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if !*failed { + *failed = true + return unavailable + } + return c.Create(ctx, obj, opts...) + }, + } + }, + want: []string{"green-pool-relay", "green-frontend"}, + }, + { + name: "a failed delete", + failOnce: func(failed *bool) interceptor.Funcs { + return interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if !*failed { + *failed = true + return unavailable + } + return c.Delete(ctx, obj, opts...) + }, + } + }, + withdraw: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a Relay whose exports are to be created or withdrawn, and fail the first such write") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + objects := []client.Object{relay, dgd} + if tc.withdraw { + objects = append(objects, + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend")) + } + failed := false + fakeClient := newExportTestClient(scheme, exportTestVersion, tc.failOnce(&failed), objects...) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) + step := func() error { + if tc.withdraw { + return r.withdrawExports(ctx, relay) + } + return r.reconcileExports(ctx, relay, dgd) + } + + t.Log("The failure is returned, for controller-runtime to retry with backoff, and is announced") + g.Expect(step()).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + g.Expect(drainEvents(recorder)).To(gomega.ContainElement(gomega.HavePrefix(corev1.EventTypeWarning + " " + reasonExportFailed))) + + t.Log("The retry runs once the API server answers: the exports converge with no other change") + g.Expect(step()).To(gomega.Succeed()) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))).To(gomega.ConsistOf(tc.want)) + }) + } +} + +func TestPoolExportsAreRepaired(t *testing.T) { + for _, tc := range []struct { + name string + drift func(export *unstructured.Unstructured) + }{ + { + name: "no DGD label", + drift: func(export *unstructured.Unstructured) { + labels := export.GetLabels() + delete(labels, commonconsts.KubeLabelDynamoGraphDeploymentName) + export.SetLabels(labels) + }, + }, + { + name: "a DGD label naming another DGD", + drift: func(export *unstructured.Unstructured) { + labels := export.GetLabels() + labels[commonconsts.KubeLabelDynamoGraphDeploymentName] = "blue" + export.SetLabels(labels) + }, + }, + { + name: "a controller reference that no longer blocks the Relay's deletion", + drift: func(export *unstructured.Unstructured) { + references := export.GetOwnerReferences() + references[0].BlockOwnerDeletion = new(false) + export.SetOwnerReferences(references) + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool whose frontend export, which the Relay controls, has " + tc.name + + ", and carries the MCS implementation's status and others' finalizer, label and annotation") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + intact := ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay") + changed := ownedExport(t, scheme, exportTestVersion, relay, "green-frontend") + changed.Object["status"] = map[string]any{"conditions": []any{ + map[string]any{"type": "Ready", "status": "True", "reason": "Exported", "message": ""}, + }} + changed.SetUID("export-frontend") + changed.SetFinalizers([]string{"example.com/hold"}) + changed.SetAnnotations(map[string]string{"example.com/note": "kept"}) + changed.SetLabels(map[string]string{commonconsts.KubeLabelDynamoGraphDeploymentName: dgd.Name, "team": "router"}) + tc.drift(changed) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + dgd, relay, intact.DeepCopy(), changed.DeepCopy()) + + t.Log("Reconcile: the export is repaired, and the repair is recorded") + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("RepairServiceExport")) + + t.Log("Check that it is the same export, with the controller's fields restored and everything else kept") + repaired := multicluster.NewServiceExport(exportTestVersion) + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(changed), repaired)).To(gomega.Succeed()) + g.Expect(repaired.GetUID()).To(gomega.Equal(changed.GetUID())) + g.Expect(repaired.GetLabels()).To(gomega.Equal(map[string]string{ + commonconsts.KubeLabelDynamoGraphDeploymentName: dgd.Name, "team": "router", + })) + g.Expect(metav1.GetControllerOf(repaired)).To(gomega.Equal(metav1.GetControllerOf(intact))) + g.Expect(repaired.GetAnnotations()).To(gomega.Equal(changed.GetAnnotations())) + g.Expect(repaired.GetFinalizers()).To(gomega.Equal(changed.GetFinalizers())) + g.Expect(repaired.Object["status"]).To(gomega.Equal(changed.Object["status"])) + + t.Log("Reconcile again: the export has the controller's fields, so nothing is applied or recorded") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + }) + } +} + +// staleCacheSetUp returns a pool's Relay and DGD; a cache that holds both of the +// Relay's exports as the API server held them when the cache read them, with the +// frontend export's DGD label removed; the API server, which holds the same +// exports for now; and the controller, which reads exports from the cache and +// writes them to the API server. +func staleCacheSetUp(t *testing.T) ( + *v1alpha1.DynamoPoolRelay, *v1beta1.DynamoGraphDeployment, client.WithWatch, *reconciler, *events.FakeRecorder, *runtime.Scheme, +) { + t.Helper() + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + relayExport := ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay") + relayExport.SetUID("export-relay") + frontendExport := ownedExport(t, scheme, exportTestVersion, relay, "green-frontend") + frontendExport.SetUID("export-frontend") + frontendExport.SetLabels(nil) + cache := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relayExport.DeepCopy(), frontendExport.DeepCopy()) + apiServer := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + relay, dgd, relayExport.DeepCopy(), frontendExport.DeepCopy()) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(apiServer, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) + r.exports.cache = cache + return relay, dgd, apiServer, r, recorder, scheme +} + +func TestPoolExportsAreRepairedOnlyAsTheyWereRead(t *testing.T) { + frontendKey := client.ObjectKey{Namespace: "inference", Name: "green-frontend"} + + t.Run("an export that a user took over stays theirs", func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + relay, dgd, apiServer, r, recorder, _ := staleCacheSetUp(t) + + t.Log("After the cache read the export, a user removes its controller reference") + export := multicluster.NewServiceExport(exportTestVersion) + g.Expect(apiServer.Get(ctx, frontendKey, export)).To(gomega.Succeed()) + export.SetOwnerReferences(nil) + g.Expect(apiServer.Update(ctx, export)).To(gomega.Succeed()) + + t.Log("Reconcile: the apply names the version the cache read, so the API server refuses it, which is no error") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + + t.Log("Check that the Relay did not take the export back") + g.Expect(apiServer.Get(ctx, frontendKey, export)).To(gomega.Succeed()) + g.Expect(export.GetOwnerReferences()).To(gomega.BeEmpty()) + g.Expect(export.GetLabels()).To(gomega.BeEmpty()) + }) + + t.Run("a deleted export is not created again by a repair", func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + relay, dgd, apiServer, r, recorder, _ := staleCacheSetUp(t) + + t.Log("After the cache read the export, it is deleted") + export := multicluster.NewServiceExport(exportTestVersion) + export.SetNamespace(frontendKey.Namespace) + export.SetName(frontendKey.Name) + g.Expect(apiServer.Delete(ctx, export)).To(gomega.Succeed()) + + t.Log("Reconcile: the apply names the UID the cache read, so the API server refuses it instead of creating the export") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + err := apiServer.Get(ctx, frontendKey, multicluster.NewServiceExport(exportTestVersion)) + g.Expect(apierrors.IsNotFound(err)).To(gomega.BeTrue(), "%v", err) + }) +} + +func TestPoolExportsAreWithdrawnOnlyAsTheyWereRead(t *testing.T) { + for _, tc := range []struct { + name string + // change changes the controller reference of an export, but not its UID. + change func(t *testing.T, scheme *runtime.Scheme, export *unstructured.Unstructured) + }{ + { + name: "a user removes the controller reference", + change: func(_ *testing.T, _ *runtime.Scheme, export *unstructured.Unstructured) { + export.SetOwnerReferences(nil) + }, + }, + { + name: "another Relay takes the export over", + change: func(t *testing.T, scheme *runtime.Scheme, export *unstructured.Unstructured) { + export.SetOwnerReferences(nil) + if err := controllerutil.SetControllerReference(newExportTestRelay(newExportTestDGD("blue")), export, scheme); err != nil { + t.Fatalf("set the owner of ServiceExport %s: %v", export.GetName(), err) + } + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("The cache lists both exports of a Relay that is being withdrawn") + relay, _, apiServer, r, recorder, scheme := staleCacheSetUp(t) + + t.Log("Before the DELETE, " + tc.name + " in the API server, and the export keeps its UID") + changed := multicluster.NewServiceExport(exportTestVersion) + g.Expect(apiServer.Get(ctx, client.ObjectKey{Namespace: "inference", Name: "green-frontend"}, changed)).To(gomega.Succeed()) + tc.change(t, scheme, changed) + g.Expect(apiServer.Update(ctx, changed)).To(gomega.Succeed()) + g.Expect(changed.GetUID()).To(gomega.BeEquivalentTo("export-frontend")) + + t.Log("Withdraw from the cache: the stale delete is refused, which is no error, and the Relay's other export is withdrawn") + g.Expect(r.withdrawExports(ctx, relay)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.ConsistOf( + corev1.EventTypeNormal + " DeleteServiceExport Withdrew the export of Service green-pool-relay")) + g.Expect(exportNames(listServiceExports(t, apiServer, exportTestVersion, relay.Namespace))).To(gomega.ConsistOf("green-frontend")) + + t.Log("Withdraw once the cache has the change: the Relay no longer controls the export, so it stays") + r.exports.cache = apiServer + g.Expect(r.withdrawExports(ctx, relay)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + remaining := listServiceExports(t, apiServer, exportTestVersion, relay.Namespace) + g.Expect(remaining).To(gomega.HaveLen(1)) + g.Expect(remaining[0].GetUID()).To(gomega.BeEquivalentTo("export-frontend")) + g.Expect(remaining[0].GetOwnerReferences()).To(gomega.Equal(changed.GetOwnerReferences())) + }) + } +} + +// newClusterIDProperty returns the ClusterProperty that holds the cluster's ID in +// its ClusterSet, with value. +func newClusterIDProperty(value string) *unstructured.Unstructured { + property := multicluster.NewClusterProperty() + property.SetName(multicluster.ClusterIDProperty) + property.Object["spec"] = map[string]any{"value": value} + return property +} + +func TestPoolExportsCheckTheClusterID(t *testing.T) { + for _, tc := range []struct { + name string + // watched says whether the controller watches ClusterProperties, as it does + // with a standard implementation in a cluster that serves them. + watched bool + // property is the cluster's ID in its ClusterSet, or empty without the + // property. + property string + readFails bool + wantMismatch bool + }{ + {name: "a Relay with the cluster's ID", watched: true, property: "east"}, + {name: "a Relay with another cluster ID", watched: true, property: "west", wantMismatch: true}, + {name: "a cluster without the property", watched: true}, + {name: "without the watch the check is skipped", property: "west"}, + {name: "a failed read fails the reconcile instead of skipping the check", watched: true, readFails: true}, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Create a pool of cluster east, and the cluster's ID property if the case has one") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + objects := []client.Object{dgd, relay} + if tc.property != "" { + objects = append(objects, newClusterIDProperty(tc.property)) + } + funcs := interceptor.Funcs{} + if tc.readFails { + funcs.Get = func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if obj.GetObjectKind().GroupVersionKind() == multicluster.ClusterPropertyGVK { + return apierrors.NewForbidden(schema.GroupResource{Group: "about.k8s.io", Resource: "clusterproperties"}, key.Name, errors.New("denied")) + } + return c.Get(ctx, key, obj, opts...) + } + } + fakeClient := newExportTestClient(scheme, exportTestVersion, funcs, objects...) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + r.exports.clusterPropertyWatched = tc.watched + + t.Log("Reconcile the pool's exports, and look for the mismatch warning") + err := r.reconcileExports(context.Background(), relay, dgd) + recorded := drainEvents(recorder) + if tc.readFails { + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("denied"))) + g.Expect(countReasons(recorded, reasonExportFailed)).To(gomega.Equal(1)) + return + } + g.Expect(err).NotTo(gomega.HaveOccurred()) + if !tc.wantMismatch { + g.Expect(countReasons(recorded, reasonClusterIDMismatch)).To(gomega.BeZero()) + return + } + g.Expect(recorded).To(gomega.ContainElement(gomega.SatisfyAll( + gomega.HavePrefix(corev1.EventTypeWarning+" "+reasonClusterIDMismatch), + gomega.ContainSubstring(`clusterId "east"`), + gomega.ContainSubstring(`the ID "west"`), + ))) + }) + } +} + +func TestPoolsFollowTheClusterIDProperty(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create two pools of cluster east in a cluster whose ID property says east, read through the watch's cache") + scheme := newExportTestScheme(t) + green := newExportTestDGD("green", "Frontend") + blue := newExportTestDGD("blue", "Frontend") + greenRelay, blueRelay := newExportTestRelay(green), newExportTestRelay(blue) + property := newClusterIDProperty("east") + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, green, blue, greenRelay, blueRelay, property) + recorder := events.NewFakeRecorder(20) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + r.exports.clusterPropertyWatched = true + + // mismatches reconciles the exports of the Relays of requests, and returns how + // many of them report a mismatch. + mismatches := func(requests []ctrl.Request) int { + for _, request := range requests { + relay := &v1alpha1.DynamoPoolRelay{} + g.Expect(fakeClient.Get(ctx, request.NamespacedName, relay)).To(gomega.Succeed()) + dgd := &v1beta1.DynamoGraphDeployment{} + g.Expect(fakeClient.Get(ctx, request.NamespacedName, dgd)).To(gomega.Succeed()) + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + } + return countReasons(drainEvents(recorder), reasonClusterIDMismatch) + } + pools := []ctrl.Request{ + {NamespacedName: client.ObjectKeyFromObject(greenRelay)}, + {NamespacedName: client.ObjectKeyFromObject(blueRelay)}, + } + g.Expect(mismatches(pools)).To(gomega.BeZero()) + + t.Log("The cluster is renamed west: users correct both Relays first, and their watch requeues them while the cache still says east") + for _, relay := range []*v1alpha1.DynamoPoolRelay{greenRelay, blueRelay} { + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(relay), relay)).To(gomega.Succeed()) + relay.Spec.ClusterID = "west" + g.Expect(fakeClient.Update(ctx, relay)).To(gomega.Succeed()) + } + g.Expect(mismatches(pools)).To(gomega.Equal(2)) + + // change applies update to the property in the cache, and returns the Relays + // that the resulting event requeues. + change := func(update func(property *unstructured.Unstructured)) []ctrl.Request { + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(property), property)).To(gomega.Succeed()) + before := property.DeepCopy() + update(property) + g.Expect(fakeClient.Update(ctx, property)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: before, ObjectNew: property})).To(gomega.BeTrue()) + return r.mapClusterPropertyToRelays(ctx, property) + } + setValue := func(value string) func(*unstructured.Unstructured) { + return func(property *unstructured.Unstructured) { property.Object["spec"] = map[string]any{"value": value} } + } + + t.Log("The cache sees the property's new value: its event requeues both Relays, which no longer report a mismatch") + g.Expect(mismatches(change(setValue("west")))).To(gomega.BeZero()) + + t.Log("The value changes again: both Relays report the mismatch") + g.Expect(mismatches(change(setValue("south")))).To(gomega.Equal(2)) + + t.Log("The property is deleted: its event requeues both Relays, which skip the check") + g.Expect(fakeClient.Delete(ctx, property)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Delete(event.DeleteEvent{Object: property})).To(gomega.BeTrue()) + g.Expect(mismatches(r.mapClusterPropertyToRelays(ctx, property))).To(gomega.BeZero()) + + t.Log("The property is created again with another value: its event requeues both Relays, which report the mismatch") + created := newClusterIDProperty("north") + g.Expect(fakeClient.Create(ctx, created)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Create(event.CreateEvent{Object: created})).To(gomega.BeTrue()) + g.Expect(mismatches(r.mapClusterPropertyToRelays(ctx, created))).To(gomega.Equal(2)) +} + +func TestMapServiceExportToRelays(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create pools green and one with a dot in its name, a DGD without a Relay, and a pool in another namespace") + scheme := newExportTestScheme(t) + green := newExportTestDGD("green", "Frontend", "Web") + dotted := newExportTestDGD("green3-0.6b", "Frontend") + unpooled := newExportTestDGD("blue", "Frontend") + elsewhere := newExportTestDGD("green", "Frontend") + elsewhere.Namespace = "other" + greenRelay := newExportTestRelay(green) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + green, dotted, unpooled, elsewhere, greenRelay, newExportTestRelay(dotted), newExportTestRelay(elsewhere)) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(1), configv1alpha1.MCSImplementationStandard, exportTestVersion) + + request := func(dgd *v1beta1.DynamoGraphDeployment) ctrl.Request { + return ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)} + } + for _, tc := range []struct { + name string + export *unstructured.Unstructured + want []ctrl.Request + }{ + { + name: "a user's export of a PoolRelay Service", + export: multicluster.RenderServiceExport(exportTestVersion, "inference", "green-pool-relay", "green"), + want: []ctrl.Request{request(green)}, + }, + { + name: "a frontend Service under its component's name", + export: multicluster.RenderServiceExport(exportTestVersion, "inference", "green-web", "green"), + want: []ctrl.Request{request(green)}, + }, + { + name: "the frontend Service of a DGD with a dot in its name", + export: multicluster.RenderServiceExport(exportTestVersion, "inference", "green3-0-6b-frontend", "green3-0.6b"), + want: []ctrl.Request{request(dotted)}, + }, + { + name: "an export the Relay controls for a frontend its DGD no longer has", + export: ownedExport(t, scheme, exportTestVersion, greenRelay, "green-old"), + want: []ctrl.Request{request(green)}, + }, + { + name: "a Service of a DGD without a Relay", + export: multicluster.RenderServiceExport(exportTestVersion, "inference", "blue-frontend", "blue"), + }, + { + name: "another Service", + export: multicluster.RenderServiceExport(exportTestVersion, "inference", "green", "green"), + }, + } { + t.Log("Map an event for " + tc.name + " in namespace inference to the Relays it concerns") + g.Expect(r.mapServiceExportToRelays(ctx, tc.export)).To(gomega.ConsistOf(tc.want), tc.name) + } +} + +func TestMapClusterPropertyToRelays(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Create Relays in two namespaces and one in an ephemeral namespace") + scheme := newExportTestScheme(t) + green := newExportTestRelay(newExportTestDGD("green")) + blue := newExportTestRelay(newExportTestDGD("blue")) + blue.Namespace = "team-b" + ephemeral := newExportTestRelay(newExportTestDGD("red")) + ephemeral.Namespace = "inference-ephemeral" + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, green, blue, ephemeral) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(1), configv1alpha1.MCSImplementationStandard, exportTestVersion) + + t.Log("An event of the cluster ID property requeues every Relay that the operator manages") + g.Expect(r.mapClusterPropertyToRelays(context.Background(), newClusterIDProperty("east"))).To(gomega.ConsistOf( + ctrl.Request{NamespacedName: client.ObjectKeyFromObject(green)}, + ctrl.Request{NamespacedName: client.ObjectKeyFromObject(blue)}, + )) +} + +func TestPoolExportWatchPredicates(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("An update of a user's export does not count; its creation and deletion do") + export := multicluster.RenderServiceExport(exportTestVersion, "inference", "green-frontend", "green") + statusChanged := export.DeepCopy() + statusChanged.Object["status"] = map[string]any{"conditions": []any{map[string]any{"type": "Ready", "status": "True"}}} + g.Expect(serviceExportChanged.Update(event.UpdateEvent{ObjectOld: export, ObjectNew: statusChanged})).To(gomega.BeFalse()) + g.Expect(serviceExportChanged.Create(event.CreateEvent{Object: export})).To(gomega.BeTrue()) + g.Expect(serviceExportChanged.Delete(event.DeleteEvent{Object: export})).To(gomega.BeTrue()) + + t.Log("Any update of an export with a controller counts, also one that adds or removes the controller reference") + relay := newExportTestRelay(newExportTestDGD("green")) + owned := ownedExport(t, newExportTestScheme(t), exportTestVersion, relay, "green-frontend") + unlabeled := owned.DeepCopy() + unlabeled.SetLabels(nil) + released := owned.DeepCopy() + released.SetOwnerReferences(nil) + g.Expect(serviceExportChanged.Update(event.UpdateEvent{ObjectOld: owned, ObjectNew: unlabeled})).To(gomega.BeTrue()) + g.Expect(serviceExportChanged.Update(event.UpdateEvent{ObjectOld: owned, ObjectNew: released})).To(gomega.BeTrue()) + g.Expect(serviceExportChanged.Update(event.UpdateEvent{ObjectOld: released, ObjectNew: owned})).To(gomega.BeTrue()) + + t.Log("A namespace-restricted operator still sees the cluster-scoped ClusterProperty, and filters other objects by namespace") + filter := eventFilter( + &configv1alpha1.OperatorConfiguration{Namespace: configv1alpha1.NamespaceConfiguration{Restricted: "inference"}}, + &commoncontroller.RuntimeConfig{}, + ) + g.Expect(filter.Create(event.CreateEvent{Object: newClusterIDProperty("east")})).To(gomega.BeTrue()) + g.Expect(filter.Create(event.CreateEvent{Object: export})).To(gomega.BeTrue()) + elsewhere := multicluster.RenderServiceExport(exportTestVersion, "other", "green-frontend", "green") + g.Expect(filter.Create(event.CreateEvent{Object: elsewhere})).To(gomega.BeFalse()) +} + +func TestClusterIDPropertyEvents(t *testing.T) { + g := gomega.NewGomegaWithT(t) + property := newClusterIDProperty("east") + other := newClusterIDProperty("east") + other.SetName("clusterset.k8s.io") + + t.Log("The creation and deletion of the cluster ID property count, another property's do not") + g.Expect(clusterIDPropertyEvents.Create(event.CreateEvent{Object: property})).To(gomega.BeTrue()) + g.Expect(clusterIDPropertyEvents.Delete(event.DeleteEvent{Object: property})).To(gomega.BeTrue()) + g.Expect(clusterIDPropertyEvents.Create(event.CreateEvent{Object: other})).To(gomega.BeFalse()) + g.Expect(clusterIDPropertyEvents.Delete(event.DeleteEvent{Object: other})).To(gomega.BeFalse()) + + t.Log("An update counts when it changes the cluster ID property's value") + relabeled := property.DeepCopy() + relabeled.SetLabels(map[string]string{"team": "router"}) + otherChanged := other.DeepCopy() + otherChanged.Object["spec"] = map[string]any{"value": "west"} + g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: property, ObjectNew: newClusterIDProperty("west")})).To(gomega.BeTrue()) + g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: property, ObjectNew: relabeled})).To(gomega.BeFalse()) + g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: other, ObjectNew: otherChanged})).To(gomega.BeFalse()) +} + +// failingRESTMapper fails every REST mapping, as when discovery is down. +type failingRESTMapper struct { + meta.RESTMapper +} + +func (failingRESTMapper) RESTMapping(schema.GroupKind, ...string) (*meta.RESTMapping, error) { + return nil, errors.New("discovery is down") +} + +func TestClusterPropertyServed(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("A cluster that serves the About API gets the watch") + served := meta.NewDefaultRESTMapper(nil) + served.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) + g.Expect(clusterPropertyServed(served)).To(gomega.BeTrue()) + + t.Log("Without the API, the controller starts without the watch, and so skips the check") + g.Expect(clusterPropertyServed(meta.NewDefaultRESTMapper(nil))).To(gomega.BeFalse()) + + t.Log("A failed discovery fails the start") + _, err := clusterPropertyServed(failingRESTMapper{RESTMapper: meta.NewDefaultRESTMapper(nil)}) + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("discovery is down"))) +} diff --git a/deploy/operator/internal/controller/setup.go b/deploy/operator/internal/controller/setup.go index d3b586a3e941..56355c42473d 100644 --- a/deploy/operator/internal/controller/setup.go +++ b/deploy/operator/internal/controller/setup.go @@ -102,9 +102,6 @@ func SetupDynamoGraphDeployment(mgr ctrl.Manager, opts DynamoGraphDeploymentSetu RestConfig: mgr.GetConfig(), DockerSecretRetriever: opts.DockerSecretRetriever, SSHKeyManager: opts.SSHKeyManager, - APIReader: mgr.GetAPIReader(), - Cache: mgr.GetCache(), - clusterSetIDs: &clusterSetIDs{}, }).SetupWithManager(mgr); err != nil { return fmt.Errorf("unable to create DynamoGraphDeployment controller: %w", err) } diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 6282056a2dae..74a1a30c60db 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -5,7 +5,7 @@ // Package multicluster renders the ServiceExports that announce a Global Router // pool to other clusters through the Kubernetes Multi-Cluster Services (MCS) API, -// and reads what the MCS implementation reports about them. +// lists them, and reads the cluster's ID in its ClusterSet. package multicluster import ( @@ -22,7 +22,6 @@ import ( "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/validation" @@ -168,47 +167,33 @@ func RenderServiceExport(version, namespace, name, dgdName string) *unstructured return serviceExport } -// ExportState is what the MCS implementation reports about a ServiceExport. -type ExportState int - -const ( - // ExportPending means that the import is not finished, or that a standard - // implementation has not reported on the export yet. - ExportPending ExportState = iota - // ExportAccepted means that nothing is reported against the export. Karmada - // writes no conditions, so its exports are always accepted. - ExportAccepted - // ExportConflicting means that some property of the export conflicts with - // another cluster's export of the Service. The implementation resolves the - // conflict, so the import can still work. - ExportConflicting - // ExportRejected means that Valid is False, or Ready is False for a reason - // other than Pending. - ExportRejected -) - // Export is a ServiceExport as the operator sees it. type Export struct { Name string UID types.UID - // ControllerUID is the UID of the export's controller, empty without one. - ControllerUID types.UID + // ResourceVersion is the version of the export that was read. A write + // conditioned on it fails once the export has changed. + ResourceVersion string + // Labels are the export's labels, and Controller its controller reference, + // nil without one: the operator writes both on the exports of a pool. + Labels map[string]string + Controller *metav1.OwnerReference // Deleting is true once the export's deletion has started. Deleting bool - // State is what the MCS implementation reports on the export, and Detail the - // condition that decided it, for people. - State ExportState - Detail string } -// ListExports returns the ServiceExports in namespace by name, read at version, -// with what implementation reports on them. reader should serve unstructured -// objects from a cache: the manager's client reads them from the API server. +// ControlledBy reports whether the object with uid controls the export. +func (e Export) ControlledBy(uid types.UID) bool { + return e.Controller != nil && e.Controller.UID == uid +} + +// ListExports returns the ServiceExports in namespace by name, read at version. +// reader should serve unstructured objects from a cache: the manager's client +// reads them from the API server. func ListExports( ctx context.Context, reader client.Reader, version, namespace string, - implementation configv1alpha1.MCSImplementation, ) (map[string]Export, error) { list := &unstructured.UnstructuredList{} list.SetGroupVersionKind(schema.GroupVersionKind{ @@ -220,83 +205,46 @@ func ListExports( exports := make(map[string]Export, len(list.Items)) for i := range list.Items { item := &list.Items[i] - export := Export{Name: item.GetName(), UID: item.GetUID(), Deleting: item.GetDeletionTimestamp() != nil} - if controller := metav1.GetControllerOf(item); controller != nil { - export.ControllerUID = controller.UID + export := Export{ + Name: item.GetName(), + UID: item.GetUID(), + ResourceVersion: item.GetResourceVersion(), + Labels: item.GetLabels(), + Controller: metav1.GetControllerOf(item), + Deleting: item.GetDeletionTimestamp() != nil, } - export.State, export.Detail = ReadExportState(implementation, item) exports[export.Name] = export } return exports, nil } -// ReadExportState reads the conditions that implementation writes on a -// ServiceExport (KEP-1645: Valid, Ready and Conflict), and returns the export's -// state with the condition that decided it, for people. -func ReadExportState( - implementation configv1alpha1.MCSImplementation, - serviceExport *unstructured.Unstructured, -) (ExportState, string) { - var status struct { - Conditions []metav1.Condition `json:"conditions"` - } - if fields, ok := serviceExport.Object["status"].(map[string]any); ok { - // A status that cannot be read counts as no report. - _ = runtime.DefaultUnstructuredConverter.FromUnstructured(fields, &status) - } - - if valid := meta.FindStatusCondition(status.Conditions, "Valid"); valid != nil && - valid.Status == metav1.ConditionFalse { - return ExportRejected, describeCondition(valid) - } - // Pending is a reason for Ready to be False or Unknown while the export - // is in progress. - if ready := meta.FindStatusCondition(status.Conditions, "Ready"); ready != nil { - switch { - case ready.Status == metav1.ConditionFalse && ready.Reason != "Pending": - return ExportRejected, describeCondition(ready) - case ready.Status != metav1.ConditionTrue: - return ExportPending, describeCondition(ready) - } - } - if conflict := meta.FindStatusCondition(status.Conditions, "Conflict"); conflict != nil && - conflict.Status == metav1.ConditionTrue { - return ExportConflicting, describeCondition(conflict) - } - if len(status.Conditions) == 0 && implementation != configv1alpha1.MCSImplementationKarmada { - return ExportPending, "no conditions yet" - } - return ExportAccepted, "" +// NewClusterProperty returns an empty ClusterProperty, to watch properties or to +// read one. +func NewClusterProperty() *unstructured.Unstructured { + property := &unstructured.Unstructured{} + property.SetGroupVersionKind(ClusterPropertyGVK) + return property } -// describeCondition returns condition as ` is (): `, -// leaving out a missing reason or message. -func describeCondition(condition *metav1.Condition) string { - description := fmt.Sprintf("%s is %s", condition.Type, condition.Status) - if condition.Reason != "" { - description += fmt.Sprintf(" (%s)", condition.Reason) - } - if condition.Message != "" { - description += ": " + condition.Message - } - return description +// ClusterPropertyValue returns the value of property, or "" without one. +func ClusterPropertyValue(property *unstructured.Unstructured) string { + value, _, _ := unstructured.NestedString(property.Object, "spec", "value") + return value } // ReadClusterID returns the cluster's ID in its ClusterSet, from the About API's // ClusterProperty named ClusterIDProperty, or "" when the cluster has no such -// property or the reader may not read it, as in a namespace-restricted install. -// reader should bypass the cache: the property is cluster-scoped, and reading it -// is optional. +// property or does not serve the About API. Any other failure, such as a missing +// permission, is an error. reader should be the manager's cache, which a watch of +// ClusterProperties fills. func ReadClusterID(ctx context.Context, reader client.Reader) (string, error) { - property := &unstructured.Unstructured{} - property.SetGroupVersionKind(ClusterPropertyGVK) + property := NewClusterProperty() err := reader.Get(ctx, client.ObjectKey{Name: ClusterIDProperty}, property) - if apierrors.IsNotFound(err) || apierrors.IsForbidden(err) || meta.IsNoMatchError(err) { + if apierrors.IsNotFound(err) || meta.IsNoMatchError(err) { return "", nil } if err != nil { return "", fmt.Errorf("failed to read ClusterProperty %s: %w", ClusterIDProperty, err) } - value, _, _ := unstructured.NestedString(property.Object, "spec", "value") - return value, nil + return ClusterPropertyValue(property), nil } diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go index afc598e1ca60..4ccfdb44a94c 100644 --- a/deploy/operator/internal/multicluster/serviceexport_test.go +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -8,7 +8,6 @@ package multicluster import ( "context" "errors" - "maps" "slices" "strings" "testing" @@ -17,6 +16,7 @@ import ( configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + "k8s.io/apimachinery/pkg/api/equality" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -119,126 +119,6 @@ func TestRenderServiceExport(t *testing.T) { } } -// exportWithConditions returns a ServiceExport whose status holds conditions, -// each given as type, status, reason and message. -func exportWithConditions(conditions ...[4]string) *unstructured.Unstructured { - export := RenderServiceExport("v1alpha1", "dynamo", "qwen-pool-relay", "qwen") - items := make([]any, 0, len(conditions)) - for _, condition := range conditions { - items = append(items, map[string]any{ - "type": condition[0], "status": condition[1], "reason": condition[2], "message": condition[3], - }) - } - if len(items) > 0 { - export.Object["status"] = map[string]any{"conditions": items} - } - return export -} - -func TestReadExportState(t *testing.T) { - const ( - standard = configv1alpha1.MCSImplementationStandard - karmada = configv1alpha1.MCSImplementationKarmada - ) - for _, tc := range []struct { - name string - implementation configv1alpha1.MCSImplementation - export *unstructured.Unstructured - want ExportState - wantDetail string - }{ - {name: "Karmada writes no status", implementation: karmada, export: exportWithConditions(), want: ExportAccepted}, - { - name: "a standard implementation has not reported yet", - implementation: standard, - export: exportWithConditions(), - want: ExportPending, - wantDetail: "no conditions yet", - }, - { - name: "valid, as an implementation without Ready reports it", - implementation: standard, - export: exportWithConditions([4]string{"Valid", "True", "Valid", ""}), - want: ExportAccepted, - }, - { - name: "valid and imported", - implementation: standard, - export: exportWithConditions( - [4]string{"Valid", "True", "Valid", ""}, - [4]string{"Ready", "True", "Exported", ""}, - [4]string{"Conflict", "False", "NoConflicts", ""}), - want: ExportAccepted, - }, - { - name: "a missing Service is invalid", - implementation: standard, - export: exportWithConditions([4]string{"Valid", "False", "NoService", "Service qwen-pool-relay not found"}), - want: ExportRejected, - wantDetail: "Valid is False (NoService): Service qwen-pool-relay not found", - }, - { - name: "an export the registry refused", - implementation: standard, - export: exportWithConditions([4]string{"Ready", "False", "Failed", ""}), - want: ExportRejected, - wantDetail: "Ready is False (Failed)", - }, - { - name: "an import that has not finished, reported as Unknown", - implementation: standard, - export: exportWithConditions([4]string{"Valid", "True", "", ""}, [4]string{"Ready", "Unknown", "Pending", ""}), - want: ExportPending, - wantDetail: "Ready is Unknown (Pending)", - }, - { - name: "an import that has not finished, reported as False", - implementation: standard, - export: exportWithConditions([4]string{"Valid", "True", "", ""}, [4]string{"Ready", "False", "Pending", ""}), - want: ExportPending, - wantDetail: "Ready is False (Pending)", - }, - { - name: "a conflict that the implementation resolved", - implementation: standard, - export: exportWithConditions( - [4]string{"Valid", "True", "", ""}, - [4]string{"Ready", "True", "Exported", ""}, - [4]string{"Conflict", "True", "PortConflict", "ports differ"}), - want: ExportConflicting, - wantDetail: "Conflict is True (PortConflict): ports differ", - }, - { - name: "a conflict while the import is not finished", - implementation: standard, - export: exportWithConditions( - [4]string{"Ready", "Unknown", "Pending", ""}, - [4]string{"Conflict", "True", "PortConflict", ""}), - want: ExportPending, - wantDetail: "Ready is Unknown (Pending)", - }, - { - name: "a status that cannot be read counts as no report", - implementation: standard, - export: func() *unstructured.Unstructured { - export := exportWithConditions() - export.Object["status"] = map[string]any{"conditions": "not a list"} - return export - }(), - want: ExportPending, - wantDetail: "no conditions yet", - }, - } { - t.Run(tc.name, func(t *testing.T) { - t.Log("Read what the MCS implementation reports on the export") - got, detail := ReadExportState(tc.implementation, tc.export) - if got != tc.want || detail != tc.wantDetail { - t.Errorf("ReadExportState = %v, %q, want %v, %q", got, detail, tc.want, tc.wantDetail) - } - }) - } -} - func TestCheckServiceName(t *testing.T) { for _, tc := range []struct { name string @@ -264,11 +144,11 @@ func TestCheckServiceName(t *testing.T) { } func TestListExports(t *testing.T) { - t.Log("Store an export controlled by a DGD that is being deleted, and one a user created that Karmada reports nothing on") - owned := exportWithConditions([4]string{"Valid", "False", "NoService", ""}) + t.Log("Store an export controlled by a DynamoPoolRelay that is being deleted, and one a user created") + owned := RenderServiceExport("v1alpha1", "dynamo", "qwen-pool-relay", "qwen") owned.SetUID("export-1") owned.SetOwnerReferences([]metav1.OwnerReference{ - {APIVersion: "nvidia.com/v1beta1", Kind: "DynamoGraphDeployment", Name: "qwen", UID: "dgd-1", Controller: new(true)}, + {APIVersion: "nvidia.com/v1alpha1", Kind: "DynamoPoolRelay", Name: "qwen", UID: "relay-1", Controller: new(true)}, }) owned.SetFinalizers([]string{"example.com/hold"}) owned.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) @@ -283,19 +163,24 @@ func TestListExports(t *testing.T) { WithObjects(owned, foreign, elsewhere). Build() - t.Log("List the exports in namespace dynamo, as Karmada sees them") - got, err := ListExports(context.Background(), reader, "v1alpha1", "dynamo", configv1alpha1.MCSImplementationKarmada) + t.Log("List the exports in namespace dynamo") + got, err := ListExports(context.Background(), reader, "v1alpha1", "dynamo") if err != nil { t.Fatalf("ListExports: %v", err) } + // The fake client gives the objects it starts with resourceVersion 999. + labels := map[string]string{commonconsts.KubeLabelDynamoGraphDeploymentName: "qwen"} want := map[string]Export{ "qwen-pool-relay": { - Name: "qwen-pool-relay", UID: "export-1", ControllerUID: "dgd-1", Deleting: true, - State: ExportRejected, Detail: "Valid is False (NoService)", + Name: "qwen-pool-relay", UID: "export-1", ResourceVersion: "999", Labels: labels, + Controller: &metav1.OwnerReference{ + APIVersion: "nvidia.com/v1alpha1", Kind: "DynamoPoolRelay", Name: "qwen", UID: "relay-1", Controller: new(true), + }, + Deleting: true, }, - "qwen-frontend": {Name: "qwen-frontend", UID: "export-2", State: ExportAccepted}, + "qwen-frontend": {Name: "qwen-frontend", UID: "export-2", ResourceVersion: "999", Labels: labels}, } - if !maps.Equal(got, want) { + if !equality.Semantic.DeepEqual(got, want) { t.Errorf("ListExports = %+v, want %+v", got, want) } } @@ -320,7 +205,7 @@ func TestReadClusterID(t *testing.T) { {name: "no cluster ID property", aboutAPI: true, objects: []client.Object{otherProperty}}, {name: "no About API"}, { - name: "no permission, as in a namespace-restricted install", aboutAPI: true, + name: "no permission is an error, not a missing ID", aboutAPI: true, wantError: true, getError: apierrors.NewForbidden(schema.GroupResource{Group: "about.k8s.io", Resource: "clusterproperties"}, ClusterIDProperty, failure), }, {name: "another failure", aboutAPI: true, getError: failure, wantError: true}, diff --git a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md index 8a8b2174d29c..b1913426fe19 100644 --- a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md +++ b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md @@ -385,13 +385,37 @@ for message sizes, reconnect behavior, and error reasons. ### Export to Other Clusters with MCS When the operator is configured with a Multi-Cluster Services implementation -(`dynamo-operator.dynamo.multiCluster.mcsImplementation`), a `DynamoPoolRelay` also makes its DGD a -Global Router pool. The operator exports `-pool-relay` and the DGD's frontend Services to the -other clusters of the ClusterSet. With a standard MCS implementation, set `clusterId` to the -cluster's ID in its ClusterSet, which the About API's `ClusterProperty` `cluster.clusterset.k8s.io` -holds. The DGD's `GlobalRouterPoolExported` condition reports a mismatch when the operator can read -that property. See -[Global Router pool export](../../reference/kubernetes-api/dynamo-graph-deployment.mdx#global-router-pool-export). +(`dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), a `DynamoPoolRelay` +also makes its DGD a Global Router pool. The operator creates a `multicluster.x-k8s.io/v1alpha1` +`ServiceExport` in the DGD's namespace for the PoolRelay Service, `-pool-relay`, and for the +Service of each frontend component, `-` in lowercase with dots replaced by +dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the +pool. It finds the frontends by a Service name suffix, `-frontend` by default, which matches a +frontend component named `Frontend`. The operator does not know that suffix, so a frontend component +with another name is exported but not found until the suffix matches it. + +The `DynamoPoolRelay` owns these exports, so deleting it deletes them. They follow the relay's own +Service: the operator withdraws them whenever it withdraws that Service, for example when the DGD is +deleted or has no workers, as the relay's `Ready` condition reports, and exports a DGD created again +under the relay's name. Renaming or removing a frontend component withdraws that component's export. +The operator restores the DGD label and the owner of an export it created if someone changes them, +and leaves a `ServiceExport` of the same name that it did not create alone; once that one is +deleted, the operator creates its own. Without a configured implementation, nothing is exported. + +With a standard MCS implementation, set `clusterId` to the cluster's ID in its ClusterSet, which the +About API's `ClusterProperty` `cluster.clusterset.k8s.io` holds. + +The MCS implementation reports on each export in the export's status, with the conditions `Valid`, +`Ready` and `Conflict`; Karmada writes none. The operator records the problems that only it can see +as Warning events on the `DynamoPoolRelay`, which `kubectl describe dynamopoolrelay ` shows: + +| Reason | Meaning | +| --- | --- | +| `InvalidServiceName` | No Service can have a name that the pool needs, so that Service is not exported. For example, a DGD whose name has a dot has no PoolRelay Service. | +| `ServiceNameTooLong` | With `karmada`, a Service name is longer than 55 characters, which Karmada cannot import, so that Service is not exported. | +| `NoFrontend` | The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests. | +| `ClusterIDMismatch` | With `mcs`, the relay's `clusterId` differs from the cluster's ID in its ClusterSet. The Global Discovery Service lists the pool only when they match. The operator watches the property, so creating, changing or deleting it checks every pool again. It skips the check when the cluster does not serve the About API or has no such property. | +| `ExportFailed` | Listing, creating, repairing or deleting an export failed. The operator retries with backoff. | ## Optional mTLS Sidecar diff --git a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md index 4108eb17b321..f2de2b44feba 100644 --- a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md +++ b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md @@ -4355,7 +4355,7 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | -| `mcsImplementation` _[MCSImplementation](#mcsimplementation)_ | MCSImplementation is the MCS implementation that imports the exports. Empty,
the default, exports nothing: Global Router pools are then discovered without
Kubernetes. Clearing it leaves earlier exports in place until their DGDs are
deleted. | | | +| `mcsImplementation` _[MCSImplementation](#mcsimplementation)_ | MCSImplementation is the MCS implementation that imports the exports. Empty,
the default, exports nothing: Global Router pools are then discovered without
Kubernetes. Clearing it leaves earlier exports in place until their
DynamoPoolRelays are deleted. | | | #### NamespaceConfiguration diff --git a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx index 1c2cc111f87f..3e0b5a9b5460 100644 --- a/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx +++ b/docs/fern/pages/reference/kubernetes-api/dynamo-graph-deployment.mdx @@ -143,26 +143,6 @@ A shared LPX draft has a legacy minimum of `1` but no independent scaling group. Both fields retain their native hub/spoke conversion mapping. Converting between `v1alpha1` and `v1beta1` preserves the chosen form and does not migrate a deployment or select a strategy. -## Global Router pool export - -A DGD is a Global Router pool while a [`DynamoPoolRelay`](../../kubernetes/kv-aware-routing/pool-relay.md) with its name exists in its namespace. When the operator is configured with a Multi-Cluster Services implementation (the platform chart's `dynamo-operator.dynamo.multiCluster.mcsImplementation`, `mcs` or `karmada`), it creates a `multicluster.x-k8s.io/v1alpha1` `ServiceExport` in the DGD's namespace for the PoolRelay Service, `-pool-relay`, and for the Service of each frontend component, `-` in lowercase with dots replaced by dashes. The Global Discovery Service reads the imported EndpointSlices of those Services to list the pool. It finds the frontends by a Service name suffix, `-frontend` by default, which matches a frontend component named `Frontend`. The operator does not know that suffix, so a frontend component with another name is exported but not found until the suffix matches it. - -The DGD owns these exports: deleting the `DynamoPoolRelay` withdraws them, renaming or removing a frontend component withdraws that component's export, and deleting the DGD deletes them. A `ServiceExport` of the same name that the DGD does not own is left alone. Without a configured implementation, nothing is exported. - -The `GlobalRouterPoolExported` condition reports on a pool's exports. It never fails the DGD or holds back its workloads. A new problem is also recorded as a Warning event on the DGD. - -| Reason | Status | Meaning | -| --- | --- | --- | -| `Exported` | True | Every Service is exported, and the MCS implementation reports no problem. Karmada reports nothing on its exports. | -| `ExportConflict` | True | Every Service is exported, but the MCS implementation reports `Conflict` as True: a property of an export differs from another cluster's export of the same Service, and the implementation resolved it. The message quotes that condition. | -| `ExportPending` | Unknown | An export was just created or is waiting to be replaced, a standard implementation has not reported on it yet, or it reports `Ready` as Unknown, or as False with the reason `Pending`. | -| `ExportRejected` | False | The MCS implementation reports `Valid` as False, or `Ready` as False for a reason other than `Pending`, on an export. The message quotes that condition. | -| `InvalidServiceName` | False | No Service can have a name that the pool needs, so that Service is not exported. For example, a DGD whose name has a dot has no PoolRelay Service. | -| `ServiceNameTooLong` | False | With `karmada`, a Service name is longer than 55 characters, which Karmada cannot import, so that Service is not exported. | -| `NoFrontend` | False | The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests. | -| `ClusterIDMismatch` | False | With `mcs`, the `DynamoPoolRelay`'s `clusterId` differs from the cluster's ID in its ClusterSet, which the About API's `ClusterProperty` `cluster.clusterset.k8s.io` holds. The Global Discovery Service lists the pool only when they match. The operator reads the ID at most once a minute, and skips the check when the cluster has no such property or the operator may not read it, as in a namespace-restricted install. | -| `ExportFailed` | False | Listing, creating or deleting an export failed. The operator retries within 30 seconds, or, when the DGD's own reconcile fails, the next time it reconciles the DGD. | - ## Spec reference diff --git a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx index 88625e82a253..251ba19dcd0c 100644 --- a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx +++ b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx @@ -5954,7 +5954,7 @@ MCSImplementation is set and a DynamoPoolRelay with its DGD's name exists. **Appears in:** [OperatorConfiguration](#operatorconfiguration) -MCSImplementation is the MCS implementation that imports the exports. Empty, the default, exports nothing: Global Router pools are then discovered without Kubernetes. Clearing it leaves earlier exports in place until their DGDs are deleted. +MCSImplementation is the MCS implementation that imports the exports. Empty, the default, exports nothing: Global Router pools are then discovered without Kubernetes. Clearing it leaves earlier exports in place until their DynamoPoolRelays are deleted. See [MCSImplementation](#mcsimplementation). From e69099a281028252fdc30a44fe02f9dd10cbaec7 Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Mon, 2 Nov 2026 21:25:13 -0800 Subject: [PATCH 8/9] fix(operator): keep each pool export event apart, and test every export guard An adversarial review of the pool's exports found that their events merged. client-go's events API counts an event as a repeat of an earlier one when both have the same object, reason, action and related object, whatever their notes. So the second export that a reconcile created became a repeat of the first, and ClusterIDMismatch kept the old cluster ID after the property changed. Each event now names a related object: the export that it created, repaired or deleted, or the DGD or ClusterProperty that its problem is about, at the version read. multicluster.ReadClusterID becomes ReadClusterIDProperty, which returns the property. The review also corrected the PoolRelay page: - With Karmada, clusterId must be the cluster's member cluster name, by which the Global Discovery Service matches the pool to its imports. - InvalidServiceName comes from a frontend Service name over 63 characters. A DGD name with a dot withdraws the Relay before anything is exported, so it was the wrong example. - An export whose controller reference someone removed or replaced is no longer the Relay's: it is neither repaired nor withdrawn, nor deleted with the Relay. Breaking any guard of the exports on purpose now makes a test fail, except the delete's UID precondition, which its resourceVersion precondition already implies. New tests cover: - the watches that the controller adds at startup: the ClusterProperty watch only with standard MCS and the About API, the ServiceExport watch at the resolved version, and ClusterProperty events reaching a namespace-restricted operator; - Reconcile returning a failed export write, with no requeue result; - the gate's resolution in features.New; - exports being deleted or already gone, names that became unexportable, same-named exports in another namespace, exports that another kind controls, and a cluster ID property without a value. The retry test now refuses creates only once the Relay is Ready, when only the retries of the failed reconciles can requeue it. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- .../internal/controller/poolrelay/exports.go | 30 +- .../poolrelay/exports_envtest_test.go | 265 ++++++++++++++--- .../controller/poolrelay/exports_test.go | 270 +++++++++++++++++- .../internal/features/multicluster_test.go | 64 +++++ .../internal/multicluster/serviceexport.go | 18 +- .../multicluster/serviceexport_test.go | 16 +- .../kubernetes/kv-aware-routing/pool-relay.md | 21 +- 7 files changed, 609 insertions(+), 75 deletions(-) diff --git a/deploy/operator/internal/controller/poolrelay/exports.go b/deploy/operator/internal/controller/poolrelay/exports.go index cd9b997f4db6..3e2f4315ba45 100644 --- a/deploy/operator/internal/controller/poolrelay/exports.go +++ b/deploy/operator/internal/controller/poolrelay/exports.go @@ -52,6 +52,13 @@ const exportFieldManager = "dynamo-pool-relay" // pool's exports. Apart from a failure, they report the problems that only the // operator can see; the MCS implementation reports on each export in the export's // own status. +// +// Each event that the controller records on a Relay names a related object: the +// export it created, repaired or deleted, or the DGD or ClusterProperty that a +// problem is about, at the version read. The events API counts an event with the +// same object, reason, action and related object as a repeat, whatever its note, +// so without one the events of a Relay's two exports would merge into one, and a +// problem with new details would keep its old note. const ( reasonInvalidServiceName = "InvalidServiceName" reasonServiceNameTooLong = "ServiceNameTooLong" @@ -275,7 +282,7 @@ func (r *reconciler) createExport( } return fmt.Errorf("failed to create ServiceExport %s: %w", name, err) } - r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "CreateServiceExport", "Create", + r.exports.recorder.Eventf(relay, serviceExport, corev1.EventTypeNormal, "CreateServiceExport", "Create", "Exported Service %s through MCS implementation %s", name, r.exports.config.MultiCluster.MCSImplementation) return nil } @@ -315,7 +322,7 @@ func (r *reconciler) repairExport( if err != nil { return fmt.Errorf("failed to repair ServiceExport %s: %w", current.Name, err) } - r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "RepairServiceExport", "Apply", + r.exports.recorder.Eventf(relay, desired, corev1.EventTypeNormal, "RepairServiceExport", "Apply", "Restored the DGD label and controller reference of the export of Service %s", current.Name) return nil } @@ -356,7 +363,10 @@ func (r *reconciler) withdrawExport(ctx context.Context, relay *v1alpha1.DynamoP } return fmt.Errorf("failed to delete ServiceExport %s: %w", export.Name, err) } - r.exports.recorder.Eventf(relay, nil, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", + // The event names the export deleted. + serviceExport.SetUID(export.UID) + serviceExport.SetResourceVersion(export.ResourceVersion) + r.exports.recorder.Eventf(relay, serviceExport, corev1.EventTypeNormal, "DeleteServiceExport", "Delete", "Withdrew the export of Service %s", export.Name) return nil } @@ -373,15 +383,15 @@ func (r *reconciler) reportExportProblems( ) error { recorder := r.exports.recorder if names, detail := unexportedNames(unexported, multicluster.NameInvalid); len(names) > 0 { - recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonInvalidServiceName, "Export", + recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonInvalidServiceName, "Export", "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail) } if names, detail := unexportedNames(unexported, multicluster.NameTooLong); len(names) > 0 { - recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonServiceNameTooLong, "Export", + recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonServiceNameTooLong, "Export", "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", ")) } if len(multicluster.FrontendServiceNames(dgd)) == 0 { - recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonNoFrontend, "Export", + recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonNoFrontend, "Export", "DGD %s has no frontend component, so the Global Router has nowhere to send the pool's requests", dgd.Name) } @@ -391,12 +401,12 @@ func (r *reconciler) reportExportProblems( if !r.exports.clusterPropertyWatched { return nil } - clusterID, err := multicluster.ReadClusterID(ctx, r.exports.cache) - if err != nil { + property, err := multicluster.ReadClusterIDProperty(ctx, r.exports.cache) + if err != nil || property == nil { return err } - if clusterID != "" && clusterID != relay.Spec.ClusterID { - recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonClusterIDMismatch, "Export", + if clusterID := multicluster.ClusterPropertyValue(property); clusterID != "" && clusterID != relay.Spec.ClusterID { + recorder.Eventf(relay, property, corev1.EventTypeWarning, reasonClusterIDMismatch, "Export", "clusterId %q differs from the ID %q that ClusterProperty %s gives this cluster in its ClusterSet; "+ "the Global Discovery Service lists the pool only when they match", relay.Spec.ClusterID, clusterID, multicluster.ClusterIDProperty) diff --git a/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go index c3fa301c20b0..8ed4cf087520 100644 --- a/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go +++ b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go @@ -26,7 +26,9 @@ import ( commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" "github.com/ai-dynamo/dynamo/deploy/operator/internal/features" "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" + appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" + eventsv1 "k8s.io/api/events/v1" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" @@ -34,8 +36,10 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/rest" + "k8s.io/client-go/util/retry" "k8s.io/utils/ptr" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/cache" "sigs.k8s.io/controller-runtime/pkg/client" ctrlconfig "sigs.k8s.io/controller-runtime/pkg/config" "sigs.k8s.io/controller-runtime/pkg/envtest" @@ -44,9 +48,9 @@ import ( ) // testCRD returns a CustomResourceDefinition that serves kind in group at -// v1alpha1 and keeps any field: a stand-in for the MCS and About APIs, whose +// version only and keeps any field: a stand-in for the MCS and About APIs, whose // CRDs the operator does not ship. -func testCRD(group, kind, plural string, scope apiextensionsv1.ResourceScope) *apiextensionsv1.CustomResourceDefinition { +func testCRD(group, kind, plural, version string, scope apiextensionsv1.ResourceScope) *apiextensionsv1.CustomResourceDefinition { return &apiextensionsv1.CustomResourceDefinition{ Name: plural + "." + group, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ @@ -56,7 +60,7 @@ func testCRD(group, kind, plural string, scope apiextensionsv1.ResourceScope) *a }, Scope: scope, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{ - Name: "v1alpha1", Served: true, Storage: true, + Name: version, Served: true, Storage: true, Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", XPreserveUnknownFields: new(true), }}, @@ -69,14 +73,14 @@ func testCRD(group, kind, plural string, scope apiextensionsv1.ResourceScope) *a } // startExportAPIServer starts an API server with the operator's CRDs and -// stand-ins for the ServiceExport and ClusterProperty APIs. It runs no garbage -// collector. -func startExportAPIServer(t *testing.T) *rest.Config { +// stand-ins for the ServiceExport API, served at serviceExportVersion only, and +// for the About API's ClusterProperty. It runs no garbage collector. +func startExportAPIServer(t *testing.T, serviceExportVersion string) *rest.Config { t.Helper() // about.k8s.io is a protected API group, so its CRD needs the approval // annotation that the About API's own CRD has. clusterProperties := testCRD(multicluster.ClusterPropertyGVK.Group, multicluster.ClusterPropertyGVK.Kind, - "clusterproperties", apiextensionsv1.ClusterScoped) + "clusterproperties", multicluster.ClusterPropertyGVK.Version, apiextensionsv1.ClusterScoped) clusterProperties.Annotations = map[string]string{ apiextensionsv1.KubeAPIApprovedAnnotation: "https://github.com/kubernetes/enhancements/pull/3084", } @@ -84,7 +88,7 @@ func startExportAPIServer(t *testing.T) *rest.Config { CRDDirectoryPaths: []string{filepath.Join("..", "..", "..", "config", "crd", "bases")}, CRDs: []*apiextensionsv1.CustomResourceDefinition{ testCRD(multicluster.ServiceExportGroupKind.Group, multicluster.ServiceExportGroupKind.Kind, - "serviceexports", apiextensionsv1.NamespaceScoped), + "serviceexports", serviceExportVersion, apiextensionsv1.NamespaceScoped), clusterProperties, }, ErrorIfCRDPathMissing: true, @@ -101,36 +105,50 @@ func startExportAPIServer(t *testing.T) *rest.Config { return config } +// relayControllerOptions configures the Relay controller that a test starts. +type relayControllerOptions struct { + // version is the multicluster.x-k8s.io version that the gate resolved: the + // one at which the API server serves ServiceExports. + version string + // restricted is the namespace that a namespace-restricted operator manages, + // or empty for an operator that manages every namespace. + restricted string + // wrap, when set, wraps the manager's client. + wrap func(client.Client) client.Client +} + // startRelayController starts a manager that runs the Relay controller with the // pool's exports on, for standard MCS, and returns a function that stops the -// manager and waits for it. wrap, when set, wraps the manager's client. -func startRelayController( - t *testing.T, - config *rest.Config, - wrap func(client.Client) client.Client, -) func() { +// manager and waits for it. +func startRelayController(t *testing.T, config *rest.Config, options relayControllerOptions) func() { t.Helper() - manager, err := ctrl.NewManager(config, ctrl.Options{ + managerOptions := ctrl.Options{ Scheme: newExportTestScheme(t), Metrics: server.Options{BindAddress: "0"}, // A test can start managers one after another, with the same controllers. Controller: ctrlconfig.Controller{SkipNameValidation: new(true)}, - NewClient: func(restConfig *rest.Config, options client.Options) (client.Client, error) { - c, err := client.New(restConfig, options) - if err != nil || wrap == nil { + NewClient: func(restConfig *rest.Config, clientOptions client.Options) (client.Client, error) { + c, err := client.New(restConfig, clientOptions) + if err != nil || options.wrap == nil { return c, err } - return wrap(c), nil + return options.wrap(c), nil }, - }) + } + if options.restricted != "" { + // As cmd/main.go configures a namespace-restricted operator's cache. + managerOptions.Cache.DefaultNamespaces = map[string]cache.Config{options.restricted: {}} + } + manager, err := ctrl.NewManager(config, managerOptions) if err != nil { t.Fatal(err) } if err := Setup(manager, &configv1alpha1.OperatorConfiguration{ Discovery: configv1alpha1.DiscoveryConfiguration{Backend: configv1alpha1.DiscoveryBackendKubernetes}, MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: configv1alpha1.MCSImplementationStandard}, + Namespace: configv1alpha1.NamespaceConfiguration{Restricted: options.restricted}, }, &commoncontroller.RuntimeConfig{ - Gate: features.Gates{MultiClusterExport: true, ServiceExportVersion: exportTestVersion}, + Gate: features.Gates{MultiClusterExport: true, ServiceExportVersion: options.version}, }); err != nil { t.Fatal(err) } @@ -290,18 +308,59 @@ func waitForDeletion(t *testing.T, c client.Client, obj client.Object) { }) } +// markRelayListenersAvailable reports the Relay's Deployment as rolled out, as +// the Deployment controller would if envtest ran it, and waits until the Relay +// is Ready. The controller requeues a Relay that is not Ready every 15 seconds; +// a Ready Relay is requeued only by watch events. +func markRelayListenersAvailable(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay) { + t.Helper() + ctx := context.Background() + key := types.NamespacedName{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + eventually(t, "the Relay's Deployment", func() (bool, error) { + err := c.Get(ctx, key, &appsv1.Deployment{}) + if apierrors.IsNotFound(err) { + return false, nil + } + return err == nil, err + }) + replicas := ptr.Deref(relay.Spec.Replicas, 2) + if err := retry.RetryOnConflict(retry.DefaultRetry, func() error { + deployment := &appsv1.Deployment{} + if err := c.Get(ctx, key, deployment); err != nil { + return err + } + deployment.Status = appsv1.DeploymentStatus{ + ObservedGeneration: deployment.Generation, + Replicas: replicas, + UpdatedReplicas: replicas, + ReadyReplicas: replicas, + AvailableReplicas: replicas, + } + return c.Status().Update(ctx, deployment) + }); err != nil { + t.Fatal(err) + } + eventually(t, "the Relay to be Ready", func() (bool, error) { + current := &v1alpha1.DynamoPoolRelay{} + if err := c.Get(ctx, client.ObjectKeyFromObject(relay), current); err != nil { + return false, err + } + return meta.IsStatusConditionTrue(current.Status.Conditions, "Ready"), nil + }) +} + //nolint:gocyclo // This test follows one Relay through the replacement of its DGD against the API server. func TestRelayControllerFollowsAReplacedDGD(t *testing.T) { if os.Getenv("KUBEBUILDER_ASSETS") == "" { t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") } ctx := context.Background() - config := startExportAPIServer(t) + config := startExportAPIServer(t, exportTestVersion) apiClient, err := client.New(config, client.Options{Scheme: newExportTestScheme(t)}) if err != nil { t.Fatal(err) } - stop := startRelayController(t, config, nil) + stop := startRelayController(t, config, relayControllerOptions{version: exportTestVersion}) t.Log("Create a DGD with a frontend component, and its DynamoPoolRelay from the checked-in sample") relay := readSampleRelay(t) @@ -389,7 +448,7 @@ func TestRelayControllerFollowsAReplacedDGD(t *testing.T) { } waitForDeletion(t, apiClient, second) createDGD(t, apiClient, relay, "Api") - startRelayController(t, config, nil) + startRelayController(t, config, relayControllerOptions{version: exportTestVersion}) waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-api") waitForRelayService(t, apiClient, relay, true) } @@ -417,19 +476,21 @@ func TestRelayControllerRetriesFailedExports(t *testing.T) { t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") } ctx := context.Background() - config := startExportAPIServer(t) + config := startExportAPIServer(t, exportTestVersion) apiClient, err := client.New(config, client.Options{Scheme: newExportTestScheme(t)}) if err != nil { t.Fatal(err) } failing := &failingCreates{} - failing.remaining.Store(2) - startRelayController(t, config, func(c client.Client) client.Client { - failing.Client = c - return failing + startRelayController(t, config, relayControllerOptions{ + version: exportTestVersion, + wrap: func(c client.Client) client.Client { + failing.Client = c + return failing + }, }) - t.Log("Create a pool whose DGD reports a terminal workload failure, while the API server refuses the first exports") + t.Log("Create a pool whose DGD reports a terminal workload failure, and wait until its Relay is Ready") relay := readSampleRelay(t) createNamespace(t, apiClient, relay.Namespace) dgd := createDGD(t, apiClient, relay, "Frontend") @@ -437,10 +498,148 @@ func TestRelayControllerRetriesFailedExports(t *testing.T) { if err := apiClient.Create(ctx, relay); err != nil { t.Fatal(err) } + frontendExport := relay.Name + "-frontend" + waitForExports(t, apiClient, relay, serviceName(relay.Name), frontendExport) + markRelayListenersAvailable(t, apiClient, relay) - t.Log("The controller retries on its own, and the exports converge with no other change") - waitForExports(t, apiClient, relay, serviceName(relay.Name), relay.Name+"-frontend") + t.Log("The API server refuses the next two ServiceExport creates, and a user deletes the frontend export") + failing.attempts.Store(0) + failing.remaining.Store(2) + deleted := exportsByName(t, apiClient, relay.Namespace)[frontendExport] + if err := apiClient.Delete(ctx, deleted); err != nil { + t.Fatal(err) + } + + t.Log("Only the retries of the failed reconciles requeue the Ready Relay: the export is created again, with no other change") + waitForExports(t, apiClient, relay, serviceName(relay.Name), frontendExport) + if exportsByName(t, apiClient, relay.Namespace)[frontendExport].GetUID() == deleted.GetUID() { + t.Fatal("the frontend export was not created again") + } if attempts := failing.attempts.Load(); attempts <= 2 { - t.Fatalf("the exports converged after %d create attempts, without retrying the 2 refused ones", attempts) + t.Fatalf("the export was created after %d attempts, without retrying the 2 refused ones", attempts) + } +} + +func TestRelayControllerWorksAtTheResolvedVersion(t *testing.T) { + if os.Getenv("KUBEBUILDER_ASSETS") == "" { + t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") + } + ctx := context.Background() + // A version other than the one the other tests' clusters serve. + const version = "v1beta1" + config := startExportAPIServer(t, version) + apiClient, err := client.New(config, client.Options{Scheme: newExportTestScheme(t)}) + if err != nil { + t.Fatal(err) + } + startRelayController(t, config, relayControllerOptions{version: version}) + + t.Log("Create a pool in a cluster that serves ServiceExports only at " + version) + relay := readSampleRelay(t) + createNamespace(t, apiClient, relay.Namespace) + createDGD(t, apiClient, relay, "Frontend") + if err := apiClient.Create(ctx, relay); err != nil { + t.Fatal(err) + } + + t.Log("The controller watches and writes ServiceExports at that version, so it exports the pool") + poolRelayExport := func() *unstructured.Unstructured { + items := listServiceExports(t, apiClient, version, relay.Namespace) + for i := range items { + if items[i].GetName() == serviceName(relay.Name) && ownedByRelay(&items[i], relay) { + return &items[i] + } + } + return nil + } + eventually(t, "the PoolRelay export at "+version, func() (bool, error) { + return poolRelayExport() != nil, nil + }) + markRelayListenersAvailable(t, apiClient, relay) + + t.Log("Delete the export: only the watch at " + version + " requeues the Ready Relay, which exports its Service again") + deleted := poolRelayExport() + if err := apiClient.Delete(ctx, deleted); err != nil { + t.Fatal(err) + } + eventually(t, "a new PoolRelay export", func() (bool, error) { + export := poolRelayExport() + return export != nil && export.GetUID() != deleted.GetUID(), nil + }) +} + +func TestNamespaceRestrictedRelayControllerFollowsTheClusterID(t *testing.T) { + if os.Getenv("KUBEBUILDER_ASSETS") == "" { + t.Skip("KUBEBUILDER_ASSETS is needed for the controller API test") + } + ctx := context.Background() + config := startExportAPIServer(t, exportTestVersion) + scheme := newExportTestScheme(t) + if err := eventsv1.AddToScheme(scheme); err != nil { + t.Fatal(err) + } + apiClient, err := client.New(config, client.Options{Scheme: scheme}) + if err != nil { + t.Fatal(err) + } + relay := readSampleRelay(t) + startRelayController(t, config, relayControllerOptions{version: exportTestVersion, restricted: relay.Namespace}) + + t.Log("Create a pool in the namespace that a namespace-restricted operator manages, in a cluster with no ID in its ClusterSet") + createNamespace(t, apiClient, relay.Namespace) + createDGD(t, apiClient, relay, "Frontend") + if err := apiClient.Create(ctx, relay); err != nil { + t.Fatal(err) + } + waitForExports(t, apiClient, relay, serviceName(relay.Name), relay.Name+"-frontend") + + t.Log("Each export's creation is an event of its own, though both events regard the Relay at one version") + eventually(t, "a CreateServiceExport event for each export", func() (bool, error) { + notes := strings.Join(relayEventNotes(t, apiClient, relay, "CreateServiceExport"), "\n") + return strings.Contains(notes, "Service "+serviceName(relay.Name)+" ") && + strings.Contains(notes, "Service "+relay.Name+"-frontend "), nil + }) + markRelayListenersAvailable(t, apiClient, relay) + + t.Log("Give the cluster an ID in its ClusterSet that differs from the Relay's clusterId") + property := newClusterIDProperty(relay.Spec.ClusterID + "-east") + if err := apiClient.Create(ctx, property); err != nil { + t.Fatal(err) + } + mismatchWith := func(clusterID string) func() (bool, error) { + return func() (bool, error) { + for _, note := range relayEventNotes(t, apiClient, relay, reasonClusterIDMismatch) { + if strings.Contains(note, fmt.Sprintf("ID %q", clusterID)) { + return true, nil + } + } + return false, nil + } + } + + t.Log("The cluster-scoped property's event reaches the restricted controller, which reports the mismatch on the Relay") + eventually(t, "a ClusterIDMismatch event on the Relay", mismatchWith(relay.Spec.ClusterID+"-east")) + + t.Log("Change the cluster's ID: the new mismatch is an event of its own, not a repeat of the old one") + property.Object["spec"] = map[string]any{"value": relay.Spec.ClusterID + "-west"} + if err := apiClient.Update(ctx, property); err != nil { + t.Fatal(err) + } + eventually(t, "a ClusterIDMismatch event with the new ID", mismatchWith(relay.Spec.ClusterID+"-west")) +} + +// relayEventNotes returns the notes of the events with reason that regard relay. +func relayEventNotes(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, reason string) []string { + t.Helper() + recorded := &eventsv1.EventList{} + if err := c.List(context.Background(), recorded, client.InNamespace(relay.Namespace)); err != nil { + t.Fatal(err) + } + var notes []string + for i := range recorded.Items { + if recorded.Items[i].Reason == reason && recorded.Items[i].Regarding.Name == relay.Name { + notes = append(notes, recorded.Items[i].Note) + } } + return notes } diff --git a/deploy/operator/internal/controller/poolrelay/exports_test.go b/deploy/operator/internal/controller/poolrelay/exports_test.go index 08f057d12835..d3897adf7aff 100644 --- a/deploy/operator/internal/controller/poolrelay/exports_test.go +++ b/deploy/operator/internal/controller/poolrelay/exports_test.go @@ -8,6 +8,7 @@ package poolrelay import ( "context" "errors" + "net/http" "strings" "testing" "time" @@ -30,6 +31,7 @@ import ( "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/uuid" + "k8s.io/client-go/rest" "k8s.io/client-go/tools/events" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -37,6 +39,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/interceptor" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/metrics/server" ) // exportTestVersion is the multicluster.x-k8s.io version that the tests' clusters @@ -217,7 +220,9 @@ func TestReconcileExports(t *testing.T) { foreign []string // ownedByAnother are exports that another Relay controls. ownedByAnother []string - wantOwned []string + // elsewhere are a user's exports in namespace other. + elsewhere []string + wantOwned []string // wantReasons are the reasons of the events of the first reconcile. The // second records only its warnings again. wantReasons []string @@ -252,11 +257,13 @@ func TestReconcileExports(t *testing.T) { wantReasons: []string{"CreateServiceExport", reasonNoFrontend}, }, { - name: "a DGD with a dot in its name has no PoolRelay Service to export", - dgdName: "green3-0.6b", frontends: []string{"Frontend"}, + // A Service name has at most 63 characters, and the operator does not + // shorten the names of the frontends' Services. + name: "a frontend Service name that no Service can have is not exported", + dgdName: "green", frontends: []string{"Frontend", strings.Repeat("w", 60)}, implementation: configv1alpha1.MCSImplementationKarmada, - wantOwned: []string{"green3-0-6b-frontend"}, - wantReasons: []string{"CreateServiceExport", reasonInvalidServiceName}, + wantOwned: []string{"green-pool-relay", "green-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport", reasonInvalidServiceName}, }, { name: "an export a user created is neither adopted nor deleted", @@ -288,6 +295,22 @@ func TestReconcileExports(t *testing.T) { wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, }, + { + name: "Karmada withdraws the Relay's export of a Service name it cannot import", + dgdName: longName, frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + ownedBefore: []string{longName + "-pool-relay", longName + "-frontend"}, + wantOwned: []string{longName + "-frontend"}, + wantReasons: []string{"DeleteServiceExport", reasonServiceNameTooLong}, + }, + { + name: "exports of the same names in another namespace do not count", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + elsewhere: []string{"green-pool-relay", "green-frontend"}, + wantOwned: []string{"green-pool-relay", "green-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + }, } { t.Run(tc.name, func(t *testing.T) { g := gomega.NewGomegaWithT(t) @@ -308,6 +331,9 @@ func TestReconcileExports(t *testing.T) { for _, name := range tc.ownedByAnother { objects = append(objects, ownedExport(t, scheme, exportTestVersion, another, name)) } + for _, name := range tc.elsewhere { + objects = append(objects, multicluster.RenderServiceExport(exportTestVersion, "other", name, dgd.Name)) + } fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, objects...) t.Log("Reconcile the pool's exports, as the Relay controller does when it publishes the Relay") @@ -339,6 +365,12 @@ func TestReconcileExports(t *testing.T) { } g.Expect(owned).To(gomega.ConsistOf(tc.wantOwned)) g.Expect(others).To(gomega.ConsistOf(append(append([]string{}, tc.foreign...), tc.ownedByAnother...))) + var elsewhere []string + for i, export := range listServiceExports(t, fakeClient, exportTestVersion, "other") { + elsewhere = append(elsewhere, export.GetName()) + g.Expect(metav1.GetControllerOf(&export)).To(gomega.BeNil(), i) + } + g.Expect(elsewhere).To(gomega.ConsistOf(tc.elsewhere)) }) } } @@ -443,6 +475,34 @@ func TestPoolExportsDoNotWaitForTheRelaysDeployment(t *testing.T) { To(gomega.ConsistOf("green-pool-relay", "green-frontend")) } +func TestRelayControllerReturnsAFailedExportWrite(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a publishable Relay while the API server refuses ServiceExports") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if obj.GetObjectKind().GroupVersionKind().GroupKind() == multicluster.ServiceExportGroupKind { + return apierrors.NewServiceUnavailable("the API server is restarting") + } + return c.Create(ctx, obj, opts...) + }, + }, relay, dgd) + recorder := events.NewFakeRecorder(20) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) + + t.Log("Reconcile: the Relay is published, and the failed export is returned for controller-runtime to retry with backoff") + result, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)}) + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + g.Expect(result).To(gomega.Equal(ctrl.Result{})) + key := client.ObjectKey{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + g.Expect(fakeClient.Get(ctx, key, &appsv1.Deployment{})).To(gomega.Succeed()) + g.Expect(countReasons(drainEvents(recorder), reasonExportFailed)).To(gomega.Equal(1)) +} + func TestOwnershipConflictWithdrawsThePoolsExports(t *testing.T) { g := gomega.NewGomegaWithT(t) ctx := context.Background() @@ -661,6 +721,32 @@ func TestPoolExportsAreRepaired(t *testing.T) { } } +func TestPoolExportsBeingDeletedAreLeftAlone(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool whose frontend export, which the Relay controls, has no DGD label and is being deleted") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + terminating := ownedExport(t, scheme, exportTestVersion, relay, "green-frontend") + terminating.SetLabels(nil) + terminating.SetFinalizers([]string{"example.com/hold"}) + terminating.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, dgd, relay, + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), terminating.DeepCopy()) + + t.Log("Reconcile: while its deletion is pending, the export is neither repaired nor created again") + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + current := multicluster.NewServiceExport(exportTestVersion) + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(terminating), current)).To(gomega.Succeed()) + g.Expect(current.GetLabels()).To(gomega.BeEmpty()) + g.Expect(current.GetDeletionTimestamp()).NotTo(gomega.BeNil()) +} + // staleCacheSetUp returns a pool's Relay and DGD; a cache that holds both of the // Relay's exports as the API server held them when the cache read them, with the // frontend export's DGD label removed; the API server, which holds the same @@ -782,6 +868,26 @@ func TestPoolExportsAreWithdrawnOnlyAsTheyWereRead(t *testing.T) { g.Expect(remaining[0].GetOwnerReferences()).To(gomega.Equal(changed.GetOwnerReferences())) }) } + + t.Run("an export that is already gone is no error", func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("The cache lists both exports of a Relay that is being withdrawn") + relay, _, apiServer, r, recorder, _ := staleCacheSetUp(t) + + t.Log("Before the DELETE, the frontend export is deleted in the API server") + gone := multicluster.NewServiceExport(exportTestVersion) + gone.SetNamespace(relay.Namespace) + gone.SetName("green-frontend") + g.Expect(apiServer.Delete(ctx, gone)).To(gomega.Succeed()) + + t.Log("Withdraw from the cache: the missing export is no error, and the Relay's other export is withdrawn") + g.Expect(r.withdrawExports(ctx, relay)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.ConsistOf( + corev1.EventTypeNormal + " DeleteServiceExport Withdrew the export of Service green-pool-relay")) + g.Expect(listServiceExports(t, apiServer, exportTestVersion, relay.Namespace)).To(gomega.BeEmpty()) + }) } // newClusterIDProperty returns the ClusterProperty that holds the cluster's ID in @@ -801,13 +907,16 @@ func TestPoolExportsCheckTheClusterID(t *testing.T) { watched bool // property is the cluster's ID in its ClusterSet, or empty without the // property. - property string - readFails bool - wantMismatch bool + property string + // emptyProperty adds the property with no value. + emptyProperty bool + readFails bool + wantMismatch bool }{ {name: "a Relay with the cluster's ID", watched: true, property: "east"}, {name: "a Relay with another cluster ID", watched: true, property: "west", wantMismatch: true}, {name: "a cluster without the property", watched: true}, + {name: "a property without a value gives no cluster ID", watched: true, emptyProperty: true}, {name: "without the watch the check is skipped", property: "west"}, {name: "a failed read fails the reconcile instead of skipping the check", watched: true, readFails: true}, } { @@ -819,7 +928,7 @@ func TestPoolExportsCheckTheClusterID(t *testing.T) { dgd := newExportTestDGD("green", "Frontend") relay := newExportTestRelay(dgd) objects := []client.Object{dgd, relay} - if tc.property != "" { + if tc.property != "" || tc.emptyProperty { objects = append(objects, newClusterIDProperty(tc.property)) } funcs := interceptor.Funcs{} @@ -935,18 +1044,30 @@ func TestMapServiceExportToRelays(t *testing.T) { g := gomega.NewGomegaWithT(t) ctx := context.Background() - t.Log("Create pools green and one with a dot in its name, a DGD without a Relay, and a pool in another namespace") + t.Log("Create pools green and one with a dot in its name, a DGD without a Relay, a pool in another namespace, " + + "and a pool there whose frontend Service has the name of green's PoolRelay Service") scheme := newExportTestScheme(t) green := newExportTestDGD("green", "Frontend", "Web") dotted := newExportTestDGD("green3-0.6b", "Frontend") unpooled := newExportTestDGD("blue", "Frontend") elsewhere := newExportTestDGD("green", "Frontend") elsewhere.Namespace = "other" + sameServiceName := newExportTestDGD("green-pool", "Relay") + sameServiceName.Namespace = "other" greenRelay := newExportTestRelay(green) fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, - green, dotted, unpooled, elsewhere, greenRelay, newExportTestRelay(dotted), newExportTestRelay(elsewhere)) + green, dotted, unpooled, elsewhere, sameServiceName, + greenRelay, newExportTestRelay(dotted), newExportTestRelay(elsewhere), newExportTestRelay(sameServiceName)) r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(1), configv1alpha1.MCSImplementationStandard, exportTestVersion) + t.Log("Create an export that a Deployment named blue controls") + controlledByDeployment := multicluster.RenderServiceExport(exportTestVersion, "inference", "blue-cache", "blue") + if err := controllerutil.SetControllerReference( + &appsv1.Deployment{Name: "blue", Namespace: "inference", UID: uuid.NewUUID()}, controlledByDeployment, scheme, + ); err != nil { + t.Fatal(err) + } + request := func(dgd *v1beta1.DynamoGraphDeployment) ctrl.Request { return ctrl.Request{NamespacedName: client.ObjectKeyFromObject(dgd)} } @@ -983,6 +1104,10 @@ func TestMapServiceExportToRelays(t *testing.T) { name: "another Service", export: multicluster.RenderServiceExport(exportTestVersion, "inference", "green", "green"), }, + { + name: "an export that another kind controls", + export: controlledByDeployment, + }, } { t.Log("Map an event for " + tc.name + " in namespace inference to the Relays it concerns") g.Expect(r.mapServiceExportToRelays(ctx, tc.export)).To(gomega.ConsistOf(tc.want), tc.name) @@ -1088,3 +1213,126 @@ func TestClusterPropertyServed(t *testing.T) { _, err := clusterPropertyServed(failingRESTMapper{RESTMapper: meta.NewDefaultRESTMapper(nil)}) g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("discovery is down"))) } + +func TestWatchExportsChecksTheClusterIDOnlyWhereItCan(t *testing.T) { + aboutAPI := meta.NewDefaultRESTMapper(nil) + aboutAPI.Add(multicluster.ClusterPropertyGVK, meta.RESTScopeRoot) + for _, tc := range []struct { + name string + implementation configv1alpha1.MCSImplementation + mapper meta.RESTMapper + wantWatched bool + wantError string + }{ + { + name: "standard MCS in a cluster that serves the About API checks the cluster ID", + implementation: configv1alpha1.MCSImplementationStandard, mapper: aboutAPI, wantWatched: true, + }, + { + name: "Karmada names its member clusters itself, so it does not check", + implementation: configv1alpha1.MCSImplementationKarmada, mapper: aboutAPI, + }, + { + name: "without the About API, the controller starts without the check", + implementation: configv1alpha1.MCSImplementationStandard, mapper: meta.NewDefaultRESTMapper(nil), + }, + { + name: "a failed discovery fails the start", + implementation: configv1alpha1.MCSImplementationStandard, + mapper: failingRESTMapper{RESTMapper: meta.NewDefaultRESTMapper(nil)}, + wantError: "discovery is down", + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + t.Log("Create a manager whose REST mapper serves the case's APIs; the test never starts it") + scheme := newExportTestScheme(t) + manager, err := ctrl.NewManager(&rest.Config{Host: "https://127.0.0.1:1"}, ctrl.Options{ + Scheme: scheme, + Metrics: server.Options{BindAddress: "0"}, + MapperProvider: func(*rest.Config, *http.Client) (meta.RESTMapper, error) { + return tc.mapper, nil + }, + }) + g.Expect(err).NotTo(gomega.HaveOccurred()) + + t.Log("Add the watches of the pool's exports, as Setup does when the gate is on") + r := &reconciler{Client: manager.GetClient(), scheme: scheme} + _, err = r.watchExports(manager, ctrl.NewControllerManagedBy(manager).For(&v1alpha1.DynamoPoolRelay{}), + &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: tc.implementation}, + }, + &commoncontroller.RuntimeConfig{ + Gate: features.Gates{MultiClusterExport: true, ServiceExportVersion: exportTestVersion}, + }) + if tc.wantError != "" { + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring(tc.wantError))) + return + } + g.Expect(err).NotTo(gomega.HaveOccurred()) + g.Expect(r.exports.clusterPropertyWatched).To(gomega.Equal(tc.wantWatched)) + }) + } +} + +// relatedRecorder records each event's reason with the kind and name of the +// event's related object. +type relatedRecorder struct { + recorded []string +} + +func (r *relatedRecorder) Eventf(_, related runtime.Object, _, reason, _, _ string, _ ...any) { + name := "none" + switch object := related.(type) { + case *unstructured.Unstructured: + name = object.GetKind() + "/" + object.GetName() + case *v1beta1.DynamoGraphDeployment: + name = "DynamoGraphDeployment/" + object.Name + } + r.recorded = append(r.recorded, reason+" "+name) +} + +func TestPoolExportEventsNameTheirObjects(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool with a frontend whose Service no Service can be named, an export to repair and one to " + + "withdraw, in a cluster whose ID in its ClusterSet differs from the Relay's") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend", strings.Repeat("w", 60)) + relay := newExportTestRelay(dgd) + unlabeled := ownedExport(t, scheme, exportTestVersion, relay, "green-frontend") + unlabeled.SetLabels(nil) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, dgd, unlabeled, + ownedExport(t, scheme, exportTestVersion, relay, "green-old"), newClusterIDProperty("west")) + recorder := &relatedRecorder{} + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + r.exports.clusterPropertyWatched = true + + t.Log("Reconcile: each event names the export it reports, or the DGD or property that its problem is about") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(recorder.recorded).To(gomega.ConsistOf( + "CreateServiceExport ServiceExport/green-pool-relay", + "RepairServiceExport ServiceExport/green-frontend", + "DeleteServiceExport ServiceExport/green-old", + "InvalidServiceName DynamoGraphDeployment/green", + "ClusterIDMismatch ClusterProperty/"+multicluster.ClusterIDProperty, + )) + + t.Log("Remove the frontends: the withdrawal names its export, and the missing frontend the DGD") + recorder.recorded = nil + g.Expect(r.reconcileExports(ctx, relay, newExportTestDGD("green"))).To(gomega.Succeed()) + g.Expect(recorder.recorded).To(gomega.ConsistOf( + "DeleteServiceExport ServiceExport/green-frontend", + "NoFrontend DynamoGraphDeployment/green", + "ClusterIDMismatch ClusterProperty/"+multicluster.ClusterIDProperty, + )) + + t.Log("With Karmada, a Service name too long to import names the DGD too") + recorder.recorded = nil + r.exports.config.MultiCluster.MCSImplementation = configv1alpha1.MCSImplementationKarmada + long := newExportTestDGD(strings.Repeat("a", 45), "Frontend") + g.Expect(r.reconcileExports(ctx, relay, long)).To(gomega.Succeed()) + g.Expect(recorder.recorded).To(gomega.ContainElement("ServiceNameTooLong DynamoGraphDeployment/" + long.Name)) +} diff --git a/deploy/operator/internal/features/multicluster_test.go b/deploy/operator/internal/features/multicluster_test.go index d482dc5c7968..47240af05ff1 100644 --- a/deploy/operator/internal/features/multicluster_test.go +++ b/deploy/operator/internal/features/multicluster_test.go @@ -10,10 +10,74 @@ import ( "strings" "testing" + configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/rest" + ctrl "sigs.k8s.io/controller-runtime" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" ) +func TestNewResolvesMultiClusterExport(t *testing.T) { + t.Log("Serve discovery for a cluster whose only API group is multicluster.x-k8s.io, with ServiceExports at v1alpha1") + mcs := metav1.GroupVersionForDiscovery{GroupVersion: "multicluster.x-k8s.io/v1alpha1", Version: "v1alpha1"} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var response any + switch r.URL.Path { + case "/api": + response = metav1.APIVersions{Kind: "APIVersions", Versions: []string{"v1"}} + case "/apis": + response = metav1.APIGroupList{ + Kind: "APIGroupList", APIVersion: "v1", + Groups: []metav1.APIGroup{{ + Name: "multicluster.x-k8s.io", Versions: []metav1.GroupVersionForDiscovery{mcs}, PreferredVersion: mcs, + }}, + } + case "/apis/multicluster.x-k8s.io/v1alpha1": + response = metav1.APIResourceList{ + Kind: "APIResourceList", APIVersion: "v1", + GroupVersion: mcs.GroupVersion, + APIResources: []metav1.APIResource{{Name: "serviceexports", Namespaced: true, Kind: "ServiceExport"}}, + } + default: + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(response); err != nil { + t.Errorf("write discovery response: %v", err) + } + })) + defer server.Close() + manager, err := ctrl.NewManager(&rest.Config{Host: server.URL}, ctrl.Options{ + Metrics: metricsserver.Options{BindAddress: "0"}, + }) + if err != nil { + t.Fatal(err) + } + + for _, tc := range []struct { + implementation configv1alpha1.MCSImplementation + wantVersion string + }{ + {implementation: configv1alpha1.MCSImplementationNone}, + {implementation: configv1alpha1.MCSImplementationStandard, wantVersion: "v1alpha1"}, + {implementation: configv1alpha1.MCSImplementationKarmada, wantVersion: "v1alpha1"}, + } { + t.Logf("Resolve the gates with mcsImplementation %q: a configured implementation turns the gate on", tc.implementation) + gates, err := New(t.Context(), manager, &configv1alpha1.OperatorConfiguration{ + MultiCluster: configv1alpha1.MultiClusterConfiguration{MCSImplementation: tc.implementation}, + }) + if err != nil { + t.Fatalf("New() with %q: %v", tc.implementation, err) + } + wantGate := tc.implementation != configv1alpha1.MCSImplementationNone + if gates.MultiClusterExport != wantGate || gates.ServiceExportVersion != tc.wantVersion { + t.Errorf("New() with %q: multiClusterExport %v at %q, want %v at %q", + tc.implementation, gates.MultiClusterExport, gates.ServiceExportVersion, wantGate, tc.wantVersion) + } + } +} + func TestResolveMultiClusterExport(t *testing.T) { const ( mcsPath = "/apis/multicluster.x-k8s.io/v1alpha1" diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 74a1a30c60db..582a6fe89676 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -232,19 +232,19 @@ func ClusterPropertyValue(property *unstructured.Unstructured) string { return value } -// ReadClusterID returns the cluster's ID in its ClusterSet, from the About API's -// ClusterProperty named ClusterIDProperty, or "" when the cluster has no such -// property or does not serve the About API. Any other failure, such as a missing -// permission, is an error. reader should be the manager's cache, which a watch of -// ClusterProperties fills. -func ReadClusterID(ctx context.Context, reader client.Reader) (string, error) { +// ReadClusterIDProperty returns the About API's ClusterProperty named +// ClusterIDProperty, whose value is the cluster's ID in its ClusterSet, or nil +// when the cluster has no such property or does not serve the About API. Any +// other failure, such as a missing permission, is an error. reader should be the +// manager's cache, which a watch of ClusterProperties fills. +func ReadClusterIDProperty(ctx context.Context, reader client.Reader) (*unstructured.Unstructured, error) { property := NewClusterProperty() err := reader.Get(ctx, client.ObjectKey{Name: ClusterIDProperty}, property) if apierrors.IsNotFound(err) || meta.IsNoMatchError(err) { - return "", nil + return nil, nil } if err != nil { - return "", fmt.Errorf("failed to read ClusterProperty %s: %w", ClusterIDProperty, err) + return nil, fmt.Errorf("failed to read ClusterProperty %s: %w", ClusterIDProperty, err) } - return ClusterPropertyValue(property), nil + return property, nil } diff --git a/deploy/operator/internal/multicluster/serviceexport_test.go b/deploy/operator/internal/multicluster/serviceexport_test.go index 4ccfdb44a94c..a2c0b8791865 100644 --- a/deploy/operator/internal/multicluster/serviceexport_test.go +++ b/deploy/operator/internal/multicluster/serviceexport_test.go @@ -203,7 +203,11 @@ func TestReadClusterID(t *testing.T) { }{ {name: "the cluster's ID in its ClusterSet", aboutAPI: true, objects: []client.Object{property, otherProperty}, want: "west"}, {name: "no cluster ID property", aboutAPI: true, objects: []client.Object{otherProperty}}, - {name: "no About API"}, + { + // A cache cannot match a kind that the cluster does not serve. + name: "no About API", + getError: &meta.NoKindMatchError{GroupKind: ClusterPropertyGVK.GroupKind(), SearchedVersions: []string{ClusterPropertyGVK.Version}}, + }, { name: "no permission is an error, not a missing ID", aboutAPI: true, wantError: true, getError: apierrors.NewForbidden(schema.GroupResource{Group: "about.k8s.io", Resource: "clusterproperties"}, ClusterIDProperty, failure), @@ -229,10 +233,14 @@ func TestReadClusterID(t *testing.T) { WithInterceptorFuncs(funcs). Build() - t.Log("Read the cluster's ID in its ClusterSet") - got, err := ReadClusterID(context.Background(), reader) + t.Log("Read the property that holds the cluster's ID in its ClusterSet") + property, err := ReadClusterIDProperty(context.Background(), reader) + got := "" + if property != nil { + got = ClusterPropertyValue(property) + } if (err != nil) != tc.wantError || got != tc.want { - t.Errorf("ReadClusterID = %q, %v, want %q with error %v", got, err, tc.want, tc.wantError) + t.Errorf("ReadClusterIDProperty = %q, %v, want %q with error %v", got, err, tc.want, tc.wantError) } if tc.wantError && !strings.Contains(err.Error(), ClusterIDProperty) { t.Errorf("error %q does not name the property", err) diff --git a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md index b1913426fe19..65e2ec8df457 100644 --- a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md +++ b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md @@ -398,12 +398,17 @@ The `DynamoPoolRelay` owns these exports, so deleting it deletes them. They foll Service: the operator withdraws them whenever it withdraws that Service, for example when the DGD is deleted or has no workers, as the relay's `Ready` condition reports, and exports a DGD created again under the relay's name. Renaming or removing a frontend component withdraws that component's export. -The operator restores the DGD label and the owner of an export it created if someone changes them, -and leaves a `ServiceExport` of the same name that it did not create alone; once that one is -deleted, the operator creates its own. Without a configured implementation, nothing is exported. - -With a standard MCS implementation, set `clusterId` to the cluster's ID in its ClusterSet, which the -About API's `ClusterProperty` `cluster.clusterset.k8s.io` holds. +While the relay is an export's controller, the operator restores the export's DGD label and +controller reference if someone changes them. The operator leaves alone any `ServiceExport` that the +relay does not control: one that a user created, or one whose controller reference someone removed +or replaced. Such an export stays when the pool is withdrawn, and is not deleted with the relay; once +it is deleted, the operator creates its own. Without a configured implementation, nothing is +exported. + +Set `clusterId` to the name that the MCS implementation gives this cluster, because the Global +Discovery Service matches the pool's registration to its imports by that name: with standard MCS, +the cluster's ID in its ClusterSet, which the About API's `ClusterProperty` +`cluster.clusterset.k8s.io` holds; with Karmada, this cluster's member cluster name. The MCS implementation reports on each export in the export's status, with the conditions `Valid`, `Ready` and `Conflict`; Karmada writes none. The operator records the problems that only it can see @@ -411,10 +416,10 @@ as Warning events on the `DynamoPoolRelay`, which `kubectl describe dynamopoolre | Reason | Meaning | | --- | --- | -| `InvalidServiceName` | No Service can have a name that the pool needs, so that Service is not exported. For example, a DGD whose name has a dot has no PoolRelay Service. | +| `InvalidServiceName` | No Service can have a name that the pool needs, so that Service is not exported. For example, a frontend component's Service name, `-`, is longer than 63 characters. | | `ServiceNameTooLong` | With `karmada`, a Service name is longer than 55 characters, which Karmada cannot import, so that Service is not exported. | | `NoFrontend` | The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests. | -| `ClusterIDMismatch` | With `mcs`, the relay's `clusterId` differs from the cluster's ID in its ClusterSet. The Global Discovery Service lists the pool only when they match. The operator watches the property, so creating, changing or deleting it checks every pool again. It skips the check when the cluster does not serve the About API or has no such property. | +| `ClusterIDMismatch` | With `mcs`, the relay's `clusterId` differs from the cluster's ID in its ClusterSet. The Global Discovery Service lists the pool only when they match. The operator watches the property, so creating, changing or deleting it checks every pool again. It skips the check when the cluster has no such property, or did not serve the About API when the operator started. | | `ExportFailed` | Listing, creating, repairing or deleting an export failed. The operator retries with backoff. | ## Optional mTLS Sidecar From a9182c07bfb870e22c04cb8cb62009edeb4f4cca Mon Sep 17 00:00:00 2001 From: Anna Tchernych Date: Tue, 3 Nov 2026 13:20:20 -0800 Subject: [PATCH 9/9] fix(operator): export only the pool's own Services, and report on the exports in a condition A review of the pool's exports found that they could export a Service that the pool does not control, that one refused write held back all the others, and that the warnings about the pool's problems repeated on every reconcile. - The operator exports a Service only while it exists and the pool controls it: the Relay controls its PoolRelay Service, and the DGD, or one of the DGD's DCDs, each frontend's Service. Before, a Service of such a name that someone else created was exported in the pool's name, also when an earlier apply of the Relay's resources failed before the PoolRelay Service's ownership check. Such a Service is now reported as ServiceNotOwned, and a watch of Services requeues the Relays whose DGD exports a Service of its name. - Each Service's export is created, repaired or withdrawn even when another one's write fails, and the problems are still reported; the failures are joined. Before, a refused create, for example under a quota, stopped the reconcile before a stale export was withdrawn, so a renamed frontend could hold the quota forever. - The Relay's Exported condition reports on the exports: True once the pool's Services are exported, False with reason Withdrawn, or False with the first problem's reason and every problem's message. The Warning events follow the condition, one per problem when it changes, instead of one per reconcile. Since e69099a281, every status write of the DGD had made each standing warning a new Event, against the operator's rule that events be edge-based. Event notes are capped at the API's 1 KiB. - Docs: clearing mcsImplementation leaves the exports in place, ExportFailed also covers failed reads of the pool's Services and of the cluster ID, and ReadClusterIDProperty no longer says that a missing permission is a retried error: the watch then never syncs and the controller does not start. The operator logs when it cannot check clusterId. - serviceName, which only forwarded to multicluster.PoolRelayServiceName, is gone, and the tests run their steps in the test body instead of in closures. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Anna Tchernych --- .../api/v1alpha1/dynamopoolrelay_types.go | 4 + .../bases/nvidia.com_dynamopoolrelays.yaml | 4 + .../controller/poolrelay/controller.go | 8 +- .../poolrelay/controller_envtest_test.go | 5 +- .../controller/poolrelay/controller_test.go | 11 +- .../internal/controller/poolrelay/exports.go | 359 +++++++++--- .../poolrelay/exports_envtest_test.go | 79 ++- .../controller/poolrelay/exports_test.go | 551 +++++++++++++++--- .../internal/multicluster/serviceexport.go | 5 +- .../kubernetes/kv-aware-routing/pool-relay.md | 21 +- .../additional-resources/api-reference-k8s.md | 2 +- .../kubernetes-api/full-api-reference.mdx | 2 +- 12 files changed, 838 insertions(+), 213 deletions(-) diff --git a/deploy/operator/api/v1alpha1/dynamopoolrelay_types.go b/deploy/operator/api/v1alpha1/dynamopoolrelay_types.go index 4b9e1a258878..92ef38439886 100644 --- a/deploy/operator/api/v1alpha1/dynamopoolrelay_types.go +++ b/deploy/operator/api/v1alpha1/dynamopoolrelay_types.go @@ -54,6 +54,10 @@ type DynamoPoolRelayStatus struct { // Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound, // DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers, // InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict. + // With multi-cluster export configured, Conditions also includes Exported: True + // with reason Exported once the pool's Services are exported, or False with + // reason Withdrawn, ServiceNotOwned, InvalidServiceName, ServiceNameTooLong, + // NoFrontend, ClusterIDMismatch, or ExportFailed. // +optional // +listType=map // +listMapKey=type diff --git a/deploy/operator/config/crd/bases/nvidia.com_dynamopoolrelays.yaml b/deploy/operator/config/crd/bases/nvidia.com_dynamopoolrelays.yaml index cf7a7134402a..0560f79b4361 100644 --- a/deploy/operator/config/crd/bases/nvidia.com_dynamopoolrelays.yaml +++ b/deploy/operator/config/crd/bases/nvidia.com_dynamopoolrelays.yaml @@ -8211,6 +8211,10 @@ spec: Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound, DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers, InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict. + With multi-cluster export configured, Conditions also includes Exported: True + with reason Exported once the pool's Services are exported, or False with + reason Withdrawn, ServiceNotOwned, InvalidServiceName, ServiceNameTooLong, + NoFrontend, ClusterIDMismatch, or ExportFailed. items: description: Condition contains details for one aspect of the current state of this API Resource. properties: diff --git a/deploy/operator/internal/controller/poolrelay/controller.go b/deploy/operator/internal/controller/poolrelay/controller.go index a8bbee49ea2d..0c30eb32000c 100644 --- a/deploy/operator/internal/controller/poolrelay/controller.go +++ b/deploy/operator/internal/controller/poolrelay/controller.go @@ -203,7 +203,7 @@ func (r *reconciler) publish( if ready { reason, message = "ListenerAvailable", "All requested Relay listeners are available" } - if err := r.setStatus(ctx, relay, ready, reason, message, serviceName(relay.Name), available, true); err != nil { + if err := r.setStatus(ctx, relay, ready, reason, message, multicluster.PoolRelayServiceName(relay.Name), available, true); err != nil { return ctrl.Result{}, err } if !ready { @@ -273,7 +273,7 @@ func supportsPodDiscovery(dgd *v1beta1.DynamoGraphDeployment, defaultBackend con // withdraw removes the publication endpoint before reporting an unresolved source. func (r *reconciler) withdraw(ctx context.Context, relay *v1alpha1.DynamoPoolRelay, reason, message string) (ctrl.Result, error) { - name := serviceName(relay.Name) + name := multicluster.PoolRelayServiceName(relay.Name) metadata := metav1.ObjectMeta{Name: name, Namespace: relay.Namespace} var cleanupErrors []error for _, object := range []client.Object{ @@ -490,10 +490,8 @@ func sourceJSON(namespaces []string, connectionRevision string) string { return string(encoded) } -func serviceName(dgdName string) string { return multicluster.PoolRelayServiceName(dgdName) } - func render(relay *v1alpha1.DynamoPoolRelay, namespaces []string) ([]client.Object, *appsv1.Deployment, error) { - name := serviceName(relay.Name) + name := multicluster.PoolRelayServiceName(relay.Name) if problems := validation.IsDNS1035Label(name); len(problems) != 0 { return nil, nil, fmt.Errorf("DGD name %q cannot produce a valid -pool-relay Service name %q: %s", relay.Name, name, strings.Join(problems, "; ")) } diff --git a/deploy/operator/internal/controller/poolrelay/controller_envtest_test.go b/deploy/operator/internal/controller/poolrelay/controller_envtest_test.go index 213dfaaa99ad..af10ccebf586 100644 --- a/deploy/operator/internal/controller/poolrelay/controller_envtest_test.go +++ b/deploy/operator/internal/controller/poolrelay/controller_envtest_test.go @@ -18,6 +18,7 @@ import ( v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" v1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" commoncontroller "github.com/ai-dynamo/dynamo/deploy/operator/internal/controller_common" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" @@ -65,7 +66,7 @@ func TestDeploymentAPIDefaultsDoNotCauseRolloutLoop(t *testing.T) { Spec: relaySpec("east"), } r := &reconciler{Client: apiClient, scheme: scheme, defaultBackend: configv1alpha1.DiscoveryBackendKubernetes} - key := types.NamespacedName{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + key := types.NamespacedName{Namespace: relay.Namespace, Name: multicluster.PoolRelayServiceName(relay.Name)} var firstResourceVersion string for attempt := range 2 { _, wanted, err := render(relay, []string{"green-runtime"}) @@ -230,7 +231,7 @@ func TestControllerPublishesAndWithdrawsOneDGDService(t *testing.T) { if err := apiClient.Create(ctx, &relay); err != nil { t.Fatal(err) } - serviceKey := types.NamespacedName{Namespace: key.Namespace, Name: serviceName(key.Name)} + serviceKey := types.NamespacedName{Namespace: key.Namespace, Name: multicluster.PoolRelayServiceName(key.Name)} deadline := time.Now().Add(15 * time.Second) for time.Now().Before(deadline) { var service corev1.Service diff --git a/deploy/operator/internal/controller/poolrelay/controller_test.go b/deploy/operator/internal/controller/poolrelay/controller_test.go index f8fe0f0ac3ca..5aebfd9fbb5a 100644 --- a/deploy/operator/internal/controller/poolrelay/controller_test.go +++ b/deploy/operator/internal/controller/poolrelay/controller_test.go @@ -17,6 +17,7 @@ import ( v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" v1beta1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1beta1" commonconsts "github.com/ai-dynamo/dynamo/deploy/operator/internal/consts" + "github.com/ai-dynamo/dynamo/deploy/operator/internal/multicluster" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" @@ -710,7 +711,7 @@ func TestDiscoveryConversionErrorDoesNotWithdrawPublishedRelay(t *testing.T) { {ComponentName: "decode", ComponentType: v1beta1.ComponentTypeWorker}, }}, } - childKey := types.NamespacedName{Name: serviceName(key.Name), Namespace: key.Namespace} + childKey := types.NamespacedName{Name: multicluster.PoolRelayServiceName(key.Name), Namespace: key.Namespace} service := &corev1.Service{Name: childKey.Name, Namespace: childKey.Namespace} deployment := &appsv1.Deployment{Name: childKey.Name, Namespace: childKey.Namespace} for _, obj := range []client.Object{service, deployment} { @@ -794,7 +795,7 @@ func TestApplyFailureWithdrawsStaleReadiness(t *testing.T) { {ComponentName: "decode", ComponentType: v1beta1.ComponentTypeWorker}, }}, } - childKey := types.NamespacedName{Namespace: key.Namespace, Name: serviceName(key.Name)} + childKey := types.NamespacedName{Namespace: key.Namespace, Name: multicluster.PoolRelayServiceName(key.Name)} denyDeployment := false apiClient := fake.NewClientBuilder().WithScheme(scheme). WithStatusSubresource(relay, dgd, &appsv1.Deployment{}). @@ -865,7 +866,7 @@ func TestWithdrawUsesVerifiedUIDAndPreservesGenerationOnFailure(t *testing.T) { Spec: relaySpec("east"), Status: v1alpha1.DynamoPoolRelayStatus{ObservedGeneration: 2}, } - childKey := types.NamespacedName{Name: serviceName(relay.Name), Namespace: relay.Namespace} + childKey := types.NamespacedName{Name: multicluster.PoolRelayServiceName(relay.Name), Namespace: relay.Namespace} service := &corev1.Service{Name: childKey.Name, Namespace: childKey.Namespace, UID: "old-service"} if err := controllerutil.SetControllerReference(relay, service, scheme); err != nil { t.Fatal(err) @@ -905,7 +906,7 @@ func TestPrerequisiteOwnershipConflictWithdrawsRelay(t *testing.T) { } } key := types.NamespacedName{Namespace: "inference", Name: "green"} - childKey := types.NamespacedName{Namespace: key.Namespace, Name: serviceName(key.Name)} + childKey := types.NamespacedName{Namespace: key.Namespace, Name: multicluster.PoolRelayServiceName(key.Name)} relay := &v1alpha1.DynamoPoolRelay{ Name: key.Name, Namespace: key.Namespace, UID: "relay-uid", Spec: relaySpec("east"), @@ -1020,7 +1021,7 @@ func TestWithdrawCleansOwnedDeploymentAfterForeignService(t *testing.T) { ObjectMeta: relayMeta, Spec: relaySpec("east"), } - key := types.NamespacedName{Name: serviceName(relay.Name), Namespace: relay.Namespace} + key := types.NamespacedName{Name: multicluster.PoolRelayServiceName(relay.Name), Namespace: relay.Namespace} resourceMeta := metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace} foreignService := &corev1.Service{ObjectMeta: resourceMeta} deployment := &appsv1.Deployment{ObjectMeta: resourceMeta} diff --git a/deploy/operator/internal/controller/poolrelay/exports.go b/deploy/operator/internal/controller/poolrelay/exports.go index 3e2f4315ba45..5a25bdc6bac0 100644 --- a/deploy/operator/internal/controller/poolrelay/exports.go +++ b/deploy/operator/internal/controller/poolrelay/exports.go @@ -7,10 +7,12 @@ package poolrelay import ( "context" + "errors" "fmt" "maps" "slices" "strings" + "unicode/utf8" configv1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/config/v1alpha1" v1alpha1 "github.com/ai-dynamo/dynamo/deploy/operator/api/v1alpha1" @@ -23,6 +25,7 @@ import ( "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/tools/events" ctrl "sigs.k8s.io/controller-runtime" @@ -43,23 +46,32 @@ import ( // +kubebuilder:rbac:groups=multicluster.x-k8s.io,resources=serviceexports,verbs=get;list;watch;create;patch;delete // +kubebuilder:rbac:groups=about.k8s.io,resources=clusterproperties,verbs=get;list;watch +// +kubebuilder:rbac:groups=nvidia.com,resources=dynamocomponentdeployments,verbs=get;list;watch // exportFieldManager is the field manager of the controller's writes to // ServiceExports, as of its other writes. const exportFieldManager = "dynamo-pool-relay" -// Reasons of the Warning events that the controller records on a Relay about its -// pool's exports. Apart from a failure, they report the problems that only the +// The Relay's Exported condition reports on the pool's exports: True once the +// Services that the pool can export are exported, False with reason Withdrawn +// while the Relay is withdrawn, and otherwise False with the reason of the pool's +// first problem. Apart from a failure, the problems are the ones that only the // operator can see; the MCS implementation reports on each export in the export's // own status. // -// Each event that the controller records on a Relay names a related object: the -// export it created, repaired or deleted, or the DGD or ClusterProperty that a -// problem is about, at the version read. The events API counts an event with the -// same object, reason, action and related object as a repeat, whatever its note, -// so without one the events of a Relay's two exports would merge into one, and a -// problem with new details would keep its old note. +// The controller records a Warning event for each problem when the condition +// changes, not on every reconcile. A problem's event names the DGD or +// ClusterProperty that it is about, at the version read, so that a problem with +// new details is not merged into the old one: the events API counts an event +// with the same object, reason, action and related object as a repeat, whatever +// its note. Normal events record each export that the controller creates, +// repairs or deletes, and name that export. const ( + conditionExported = "Exported" + + reasonExported = "Exported" + reasonWithdrawn = "Withdrawn" + reasonServiceNotOwned = "ServiceNotOwned" reasonInvalidServiceName = "InvalidServiceName" reasonServiceNameTooLong = "ServiceNameTooLong" reasonNoFrontend = "NoFrontend" @@ -67,9 +79,19 @@ const ( reasonExportFailed = "ExportFailed" ) +// Limits of the API server: an event's note, and a condition's message. +const ( + maxEventNote = 1024 + maxConditionMessage = 32768 +) + // relayGroupKind is the kind that controls a pool's exports. var relayGroupKind = v1alpha1.GroupVersion.WithKind("DynamoPoolRelay").GroupKind() +// dcdGroupKind is the kind that controls a frontend's Service on the DCD path; a +// DGD controls each of its DCDs. +var dcdGroupKind = v1beta1.GroupVersion.WithKind("DynamoComponentDeployment").GroupKind() + // exportSettings configures the pool's exports, which the controller maintains // when the multiClusterExport gate is on. type exportSettings struct { @@ -103,9 +125,9 @@ func eventFilter( } // watchExports turns on the pool's exports, and adds the watches that they need: -// the ServiceExports and, with a standard implementation in a cluster that serves -// the About API, the ClusterProperty that holds the cluster's ID in its -// ClusterSet. The Relay's DGD is watched already. +// the ServiceExports, the Services that a pool may export and, with a standard +// implementation in a cluster that serves the About API, the ClusterProperty that +// holds the cluster's ID in its ClusterSet. The Relay's DGD is watched already. func (r *reconciler) watchExports( mgr ctrl.Manager, controllerBuilder *builder.Builder, @@ -122,14 +144,23 @@ func (r *reconciler) watchExports( multicluster.NewServiceExport(runtimeConfig.Gate.ServiceExportVersion), handler.EnqueueRequestsFromMapFunc(r.mapServiceExportToRelays), builder.WithPredicates(serviceExportChanged), + ).Watches( + &corev1.Service{}, + handler.EnqueueRequestsFromMapFunc(r.mapServiceToRelays), + builder.WithPredicates(serviceControllerChanged), ) if !multicluster.UsesClusterSetID(config.MultiCluster.MCSImplementation) { return controllerBuilder, nil } served, err := clusterPropertyServed(mgr.GetRESTMapper()) - if err != nil || !served { + if err != nil { return controllerBuilder, err } + if !served { + mgr.GetLogger().Info("The cluster does not serve the About API's ClusterProperty, " + + "so the clusterId of DynamoPoolRelays is not checked; to check it, install the About API and restart the operator") + return controllerBuilder, nil + } r.exports.clusterPropertyWatched = true return controllerBuilder.Watches( multicluster.NewClusterProperty(), @@ -138,11 +169,20 @@ func (r *reconciler) watchExports( ), nil } +// exportProblem is a problem of the pool's exports that only the operator can +// see, or a failure to reconcile them. related is the object that it is about, +// if any. +type exportProblem struct { + reason string + message string + related runtime.Object +} + // reconcileExports exports the Services of the pool that the Relay publishes for -// dgd, and withdraws the Relay's other exports. It decides from what exists now, -// not from the events that led there, so a DGD that is deleted and created again -// under the Relay's name, quickly or not, ends up with the new DGD's exports. A -// failure is also recorded as a Warning event on the Relay. +// dgd, withdraws the Relay's other exports, and reports the outcome in the +// Relay's Exported condition. It decides from what exists now, not from the +// events that led there, so a DGD that is deleted and created again under the +// Relay's name, quickly or not, ends up with the new DGD's exports. func (r *reconciler) reconcileExports( ctx context.Context, relay *v1alpha1.DynamoPoolRelay, @@ -151,26 +191,78 @@ func (r *reconciler) reconcileExports( if r.exports == nil { return nil } - return r.exportFailed(relay, r.convergeExports(ctx, relay, dgd)) + exported, missing, problems, err := r.convergeExports(ctx, relay, dgd) + message := fmt.Sprintf("Exported %s through MCS implementation %s", + serviceList(exported), r.exports.config.MultiCluster.MCSImplementation) + if len(missing) > 0 { + message += fmt.Sprintf("; %s does not exist yet", serviceList(missing)) + } + exportedCondition := metav1.Condition{ + Type: conditionExported, Status: metav1.ConditionTrue, Reason: reasonExported, Message: message, + } + return errors.Join(err, r.setExported(ctx, relay, exportedCondition, problems, err)) } // withdrawExports withdraws every export that the Relay controls, as it withdraws -// the Relay's Service. A failure is also recorded as a Warning event on the Relay. +// the Relay's Service, and reports that in the Relay's Exported condition. func (r *reconciler) withdrawExports(ctx context.Context, relay *v1alpha1.DynamoPoolRelay) error { if r.exports == nil { return nil } - return r.exportFailed(relay, r.withdrawAllExports(ctx, relay)) + err := r.withdrawAllExports(ctx, relay) + withdrawnCondition := metav1.Condition{ + Type: conditionExported, Status: metav1.ConditionFalse, Reason: reasonWithdrawn, + Message: "The Relay withdrew its Service, and with it the pool's exports", + } + return errors.Join(err, r.setExported(ctx, relay, withdrawnCondition, nil, err)) } -// exportFailed records err, a failure to reconcile the pool's exports, as a -// Warning event on the Relay, and returns it. -func (r *reconciler) exportFailed(relay *v1alpha1.DynamoPoolRelay, err error) error { +// setExported sets the Relay's Exported condition to condition or, with problems +// or err, to False with the first one's reason and every one's message. When the +// condition changes, it writes the Relay's status and then records a Warning +// event for each problem, so the events follow the condition, not each reconcile. +func (r *reconciler) setExported( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + condition metav1.Condition, + problems []exportProblem, + err error, +) error { if err != nil { - r.exports.recorder.Eventf(relay, nil, corev1.EventTypeWarning, reasonExportFailed, "Export", - "Failed to reconcile the pool's ServiceExports: %v", err) + problems = append([]exportProblem{{ + reason: reasonExportFailed, + message: fmt.Sprintf("Failed to reconcile the pool's ServiceExports: %v", err), + }}, problems...) + } + if len(problems) > 0 { + messages := make([]string, 0, len(problems)) + for _, problem := range problems { + messages = append(messages, problem.message) + } + condition.Status = metav1.ConditionFalse + condition.Reason = problems[0].reason + condition.Message = strings.Join(messages, "; ") } - return err + condition.Message = truncate(condition.Message, maxConditionMessage) + condition.ObservedGeneration = relay.Generation + + previous := meta.FindStatusCondition(relay.Status.Conditions, conditionExported) + changed := previous == nil || previous.Status != condition.Status || + previous.Reason != condition.Reason || previous.Message != condition.Message + if !changed && previous.ObservedGeneration == condition.ObservedGeneration { + return nil + } + meta.SetStatusCondition(&relay.Status.Conditions, condition) + if err := r.Status().Update(ctx, relay); err != nil { + return fmt.Errorf("failed to update the Exported condition: %w", err) + } + if changed { + for _, problem := range problems { + r.exports.recorder.Eventf(relay, problem.related, corev1.EventTypeWarning, problem.reason, "Export", + "%s", truncate(problem.message, maxEventNote)) + } + } + return nil } // unexportedService is a pool Service that the implementation cannot export. @@ -180,51 +272,115 @@ type unexportedService struct { detail string } +// convergeExports exports each Service of the pool that the implementation can +// export and that the pool controls, and withdraws the Relay's other exports. It +// returns the names exported, those whose Service does not exist yet, the pool's +// problems, and every failed read and write: one failure does not hold back the +// others. func (r *reconciler) convergeExports( ctx context.Context, relay *v1alpha1.DynamoPoolRelay, dgd *v1beta1.DynamoGraphDeployment, -) error { +) (exported, missing []string, problems []exportProblem, err error) { implementation := r.exports.config.MultiCluster.MCSImplementation existing, err := r.listExports(ctx, relay) if err != nil { - return err + return nil, nil, nil, err } - // The pool exports each of its Services that the implementation can export: a - // missing export is created, and one that the Relay controls is repaired. A - // user's export, or one being deleted, stays as it is. - var exported []string + // The pool exports each of its Services that the implementation can export and + // that the pool controls: a missing export is created, and one that the Relay + // controls is repaired. A user's export, or one being deleted, stays as it is. + // A Service that does not exist, or that the pool does not control, is not + // exported, so a Service of the same name that someone else created is never + // exported in the pool's name. + var errs []error var unexported []unexportedService + var notControlled []string + keep := map[string]bool{} for _, name := range multicluster.ServiceNames(dgd) { if problem, detail := multicluster.CheckServiceName(implementation, name); problem != multicluster.NameExportable { unexported = append(unexported, unexportedService{name: name, problem: problem, detail: detail}) continue } + found, controlled, err := r.poolControlsService(ctx, relay, dgd, name) + switch { + case err != nil: + // Keep the export until the Service can be read again. + keep[name] = true + errs = append(errs, err) + continue + case !found: + missing = append(missing, name) + continue + case !controlled: + notControlled = append(notControlled, name) + continue + } + keep[name] = true current, found := existing[name] switch { case !found: - if err := r.createExport(ctx, relay, dgd, name); err != nil { - return err - } + err = r.createExport(ctx, relay, dgd, name) case current.ControlledBy(relay.UID) && !current.Deleting: - if err := r.repairExport(ctx, relay, dgd, current); err != nil { - return err - } + err = r.repairExport(ctx, relay, dgd, current) + } + if err != nil { + errs = append(errs, err) + continue } exported = append(exported, name) } - // Withdraw the Relay's exports of Services that the pool no longer exports, - // such as a renamed or removed frontend's. + // Withdraw the Relay's exports of Services that the pool does not export, such + // as a renamed or removed frontend's, or one that someone else's Service took + // over. for _, name := range slices.Sorted(maps.Keys(existing)) { - if !slices.Contains(exported, name) { - if err := r.withdrawExport(ctx, relay, existing[name]); err != nil { - return err - } + if keep[name] { + continue + } + if err := r.withdrawExport(ctx, relay, existing[name]); err != nil { + errs = append(errs, err) } } - return r.reportExportProblems(ctx, relay, dgd, unexported) + problems, err = r.exportProblems(ctx, relay, dgd, unexported, notControlled) + return exported, missing, problems, errors.Join(append(errs, err)...) +} + +// poolControlsService reads the Service named name in the Relay's namespace, and +// reports whether it exists and whether the pool controls it: the Relay controls +// its PoolRelay Service, and the DGD, or on the DCD path one of its DCDs, a +// frontend's. +func (r *reconciler) poolControlsService( + ctx context.Context, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + name string, +) (found, controlled bool, err error) { + service := &corev1.Service{} + if err := r.Get(ctx, client.ObjectKey{Namespace: relay.Namespace, Name: name}, service); err != nil { + if apierrors.IsNotFound(err) { + return false, false, nil + } + return false, false, fmt.Errorf("failed to read Service %s: %w", name, err) + } + controller := metav1.GetControllerOf(service) + switch { + case controller == nil: + return true, false, nil + case controller.UID == relay.UID || controller.UID == dgd.UID: + return true, true, nil + case schema.FromAPIVersionAndKind(controller.APIVersion, controller.Kind).GroupKind() != dcdGroupKind: + return true, false, nil + } + dcd := &v1beta1.DynamoComponentDeployment{} + if err := r.Get(ctx, client.ObjectKey{Namespace: relay.Namespace, Name: controller.Name}, dcd); err != nil { + if apierrors.IsNotFound(err) { + return true, false, nil + } + return true, false, fmt.Errorf("failed to read DynamoComponentDeployment %s: %w", controller.Name, err) + } + return true, dcd.UID == controller.UID && metav1.IsControlledBy(dcd, dgd), nil } func (r *reconciler) withdrawAllExports(ctx context.Context, relay *v1alpha1.DynamoPoolRelay) error { @@ -232,12 +388,13 @@ func (r *reconciler) withdrawAllExports(ctx context.Context, relay *v1alpha1.Dyn if err != nil { return err } + var errs []error for _, name := range slices.Sorted(maps.Keys(existing)) { if err := r.withdrawExport(ctx, relay, existing[name]); err != nil { - return err + errs = append(errs, err) } } - return nil + return errors.Join(errs...) } // listExports returns the ServiceExports in the Relay's namespace by name. @@ -371,47 +528,76 @@ func (r *reconciler) withdrawExport(ctx context.Context, relay *v1alpha1.DynamoP return nil } -// reportExportProblems records a Warning event on the Relay for each problem of -// the pool that only the operator can see: a Service that the implementation -// cannot export, a DGD without a frontend, and a cluster ID that differs from the -// cluster's ID in its ClusterSet. A failed read of that ID is an error. -func (r *reconciler) reportExportProblems( +// exportProblems returns the problems of the pool that only the operator can +// see: a Service that the pool does not control, a Service that the +// implementation cannot export, a DGD without a frontend, and a cluster ID that +// differs from the cluster's ID in its ClusterSet. A failed read of that ID is an +// error. +func (r *reconciler) exportProblems( ctx context.Context, relay *v1alpha1.DynamoPoolRelay, dgd *v1beta1.DynamoGraphDeployment, unexported []unexportedService, -) error { - recorder := r.exports.recorder + notControlled []string, +) ([]exportProblem, error) { + var problems []exportProblem + if len(notControlled) > 0 { + problems = append(problems, exportProblem{reason: reasonServiceNotOwned, related: dgd, message: fmt.Sprintf( + "Neither the Relay nor DGD %s controls %s, so it is not exported", dgd.Name, serviceList(notControlled))}) + } if names, detail := unexportedNames(unexported, multicluster.NameInvalid); len(names) > 0 { - recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonInvalidServiceName, "Export", - "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail) + problems = append(problems, exportProblem{reason: reasonInvalidServiceName, related: dgd, message: fmt.Sprintf( + "No Service can be named %s, so it is not exported: %s", strings.Join(names, ", "), detail)}) } if names, detail := unexportedNames(unexported, multicluster.NameTooLong); len(names) > 0 { - recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonServiceNameTooLong, "Export", - "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", ")) + problems = append(problems, exportProblem{reason: reasonServiceNameTooLong, related: dgd, message: fmt.Sprintf( + "%s, so %s is not exported; shorten the DGD or component name", detail, strings.Join(names, ", "))}) } if len(multicluster.FrontendServiceNames(dgd)) == 0 { - recorder.Eventf(relay, dgd, corev1.EventTypeWarning, reasonNoFrontend, "Export", - "DGD %s has no frontend component, so the Global Router has nowhere to send the pool's requests", dgd.Name) + problems = append(problems, exportProblem{reason: reasonNoFrontend, related: dgd, message: fmt.Sprintf( + "DGD %s has no frontend component, so the Global Router has nowhere to send the pool's requests", dgd.Name)}) } // A standard implementation labels the pool's imports with the cluster's ID in // its ClusterSet, and the Global Discovery Service joins the pool's // registration to them only under the same cluster ID. if !r.exports.clusterPropertyWatched { - return nil + return problems, nil } property, err := multicluster.ReadClusterIDProperty(ctx, r.exports.cache) if err != nil || property == nil { - return err + return problems, err } if clusterID := multicluster.ClusterPropertyValue(property); clusterID != "" && clusterID != relay.Spec.ClusterID { - recorder.Eventf(relay, property, corev1.EventTypeWarning, reasonClusterIDMismatch, "Export", + problems = append(problems, exportProblem{reason: reasonClusterIDMismatch, related: property, message: fmt.Sprintf( "clusterId %q differs from the ID %q that ClusterProperty %s gives this cluster in its ClusterSet; "+ "the Global Discovery Service lists the pool only when they match", - relay.Spec.ClusterID, clusterID, multicluster.ClusterIDProperty) + relay.Spec.ClusterID, clusterID, multicluster.ClusterIDProperty)}) } - return nil + return problems, nil +} + +// serviceList names the Services named names in a message. +func serviceList(names []string) string { + switch len(names) { + case 0: + return "no Service" + case 1: + return "Service " + names[0] + } + return "Services " + strings.Join(names, ", ") +} + +// truncate shortens s to at most limit bytes without splitting a character. +func truncate(s string, limit int) string { + if len(s) <= limit { + return s + } + end := limit + for end > 0 && !utf8.RuneStart(s[end]) { + end-- + } + return s[:end] } // unexportedNames returns the names of the services with problem, and the first @@ -432,33 +618,45 @@ func unexportedNames(services []unexportedService, problem multicluster.NameProb } // mapServiceExportToRelays requeues the Relay that controls the export, and each -// Relay in the export's namespace whose DGD exports a Service of the export's -// name: such a Relay takes over when a user's export of that Service is deleted. +// Relay whose DGD exports a Service of the export's name: such a Relay takes over +// when a user's export of that Service is deleted. func (r *reconciler) mapServiceExportToRelays(ctx context.Context, export client.Object) []ctrl.Request { var requests []ctrl.Request - add := func(name string) { - request := ctrl.Request{Namespace: export.GetNamespace(), Name: name} + if controller := metav1.GetControllerOf(export); controller != nil && + schema.FromAPIVersionAndKind(controller.APIVersion, controller.Kind).GroupKind() == relayGroupKind { + requests = append(requests, ctrl.Request{Namespace: export.GetNamespace(), Name: controller.Name}) + } + for _, request := range r.relaysExportingService(ctx, export.GetNamespace(), export.GetName()) { if !slices.Contains(requests, request) { requests = append(requests, request) } } - if controller := metav1.GetControllerOf(export); controller != nil && - schema.FromAPIVersionAndKind(controller.APIVersion, controller.Kind).GroupKind() == relayGroupKind { - add(controller.Name) - } + return requests +} + +// mapServiceToRelays requeues each Relay whose DGD exports a Service of the +// Service's name: the pool exports a Service only while it exists and the pool +// controls it. +func (r *reconciler) mapServiceToRelays(ctx context.Context, service client.Object) []ctrl.Request { + return r.relaysExportingService(ctx, service.GetNamespace(), service.GetName()) +} +// relaysExportingService returns a request for each Relay in namespace whose DGD +// exports a Service named name. +func (r *reconciler) relaysExportingService(ctx context.Context, namespace, name string) []ctrl.Request { relays := &v1alpha1.DynamoPoolRelayList{} - if err := r.List(ctx, relays, client.InNamespace(export.GetNamespace())); err != nil { - log.FromContext(ctx).Error(err, "Failed to list DynamoPoolRelays for ServiceExport event") - return requests + if err := r.List(ctx, relays, client.InNamespace(namespace)); err != nil { + log.FromContext(ctx).Error(err, "Failed to list DynamoPoolRelays for a Service or ServiceExport event") + return nil } + var requests []ctrl.Request for i := range relays.Items { dgd := &v1beta1.DynamoGraphDeployment{} if err := r.Get(ctx, client.ObjectKeyFromObject(&relays.Items[i]), dgd); err != nil { continue } - if slices.Contains(multicluster.ServiceNames(dgd), export.GetName()) { - add(relays.Items[i].Name) + if slices.Contains(multicluster.ServiceNames(dgd), name) { + requests = append(requests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(&relays.Items[i])}) } } return requests @@ -492,6 +690,13 @@ var serviceExportChanged = predicate.Funcs{UpdateFunc: func(update event.UpdateE return metav1.GetControllerOf(update.ObjectOld) != nil || metav1.GetControllerOf(update.ObjectNew) != nil }} +// serviceControllerChanged passes the creation and deletion of a Service, and an +// update that changes its controller: whether the pool exports a Service depends +// only on whether it exists and who controls it. +var serviceControllerChanged = predicate.Funcs{UpdateFunc: func(update event.UpdateEvent) bool { + return !equality.Semantic.DeepEqual(metav1.GetControllerOf(update.ObjectOld), metav1.GetControllerOf(update.ObjectNew)) +}} + // clusterIDPropertyEvents passes the events of the ClusterProperty that holds the // cluster's ID in its ClusterSet: its creation, its deletion and a change of its // value. diff --git a/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go index 8ed4cf087520..19bcc201bb00 100644 --- a/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go +++ b/deploy/operator/internal/controller/poolrelay/exports_envtest_test.go @@ -42,6 +42,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/cache" "sigs.k8s.io/controller-runtime/pkg/client" ctrlconfig "sigs.k8s.io/controller-runtime/pkg/config" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" "sigs.k8s.io/controller-runtime/pkg/envtest" "sigs.k8s.io/controller-runtime/pkg/metrics/server" "sigs.k8s.io/yaml" @@ -214,7 +215,8 @@ func createNamespace(t *testing.T, c client.Client, name string) { } // createDGD creates the DGD of the Relay's name with a worker and a frontend -// component named frontend, and returns it. +// component named frontend, and the frontend's Service, controlled by the DGD as +// the DGD controller creates it, and returns the DGD. func createDGD(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, frontend string) *v1beta1.DynamoGraphDeployment { t.Helper() dgd := &v1beta1.DynamoGraphDeployment{ @@ -227,9 +229,32 @@ func createDGD(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, f if err := c.Create(context.Background(), dgd); err != nil { t.Fatal(err) } + for _, name := range multicluster.FrontendServiceNames(dgd) { + service := &corev1.Service{ + Name: name, Namespace: dgd.Namespace, + Spec: corev1.ServiceSpec{Ports: []corev1.ServicePort{{Name: "http", Port: 8000}}}, + } + if err := controllerutil.SetControllerReference(dgd, service, c.Scheme()); err != nil { + t.Fatal(err) + } + if err := c.Create(context.Background(), service); err != nil { + t.Fatal(err) + } + } return dgd } +// exportVersions returns the resourceVersion of each ServiceExport in namespace, +// by name. +func exportVersions(t *testing.T, c client.Client, namespace string) map[string]string { + t.Helper() + versions := map[string]string{} + for name, export := range exportsByName(t, c, namespace) { + versions[name] = export.GetResourceVersion() + } + return versions +} + // reportTerminalFailure writes the status that the DGD controller writes when a // DGD's workloads cannot be reconciled at all, with message. func reportTerminalFailure(t *testing.T, c client.Client, dgd *v1beta1.DynamoGraphDeployment, message string) { @@ -286,7 +311,7 @@ func waitForExports(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRel // or withdraws it. func waitForRelayService(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay, published bool) { t.Helper() - key := types.NamespacedName{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + key := types.NamespacedName{Namespace: relay.Namespace, Name: multicluster.PoolRelayServiceName(relay.Name)} eventually(t, fmt.Sprintf("the Relay's Service to be published: %v", published), func() (bool, error) { err := c.Get(context.Background(), key, &corev1.Service{}) if apierrors.IsNotFound(err) { @@ -315,7 +340,7 @@ func waitForDeletion(t *testing.T, c client.Client, obj client.Object) { func markRelayListenersAvailable(t *testing.T, c client.Client, relay *v1alpha1.DynamoPoolRelay) { t.Helper() ctx := context.Background() - key := types.NamespacedName{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + key := types.NamespacedName{Namespace: relay.Namespace, Name: multicluster.PoolRelayServiceName(relay.Name)} eventually(t, "the Relay's Deployment", func() (bool, error) { err := c.Get(ctx, key, &appsv1.Deployment{}) if apierrors.IsNotFound(err) { @@ -370,7 +395,7 @@ func TestRelayControllerFollowsAReplacedDGD(t *testing.T) { t.Fatal(err) } relayUID := relay.UID - poolRelayExport := serviceName(relay.Name) + poolRelayExport := multicluster.PoolRelayServiceName(relay.Name) waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-frontend") waitForRelayService(t, apiClient, relay, true) @@ -425,22 +450,40 @@ func TestRelayControllerFollowsAReplacedDGD(t *testing.T) { } t.Log("Status updates of the DGD, such as a terminal workload failure, write nothing to the exports") - versions := func() map[string]string { - byName := map[string]string{} - for name, export := range exportsByName(t, apiClient, relay.Namespace) { - byName[name] = export.GetResourceVersion() - } - return byName - } - before := versions() + before := exportVersions(t, apiClient, relay.Namespace) for attempt := range 3 { reportTerminalFailure(t, apiClient, second, fmt.Sprintf("Grove is disabled (report %d)", attempt)) } time.Sleep(2 * time.Second) - if after := versions(); !maps.Equal(before, after) { + if after := exportVersions(t, apiClient, relay.Namespace); !maps.Equal(before, after) { t.Fatalf("DGD status updates changed the exports: %v -> %v", before, after) } + t.Log("Only the frontend's Service changes, so only the Service watch requeues the Relay, whose listeners are " + + "available and which therefore polls nothing: deleting the Service withdraws its export, and creating it " + + "again exports it again") + markRelayListenersAvailable(t, apiClient, relay) + frontendService := &corev1.Service{} + frontendKey := client.ObjectKey{Namespace: relay.Namespace, Name: relay.Name + "-web"} + if err := apiClient.Get(ctx, frontendKey, frontendService); err != nil { + t.Fatal(err) + } + if err := apiClient.Delete(ctx, frontendService); err != nil { + t.Fatal(err) + } + waitForExports(t, apiClient, relay, poolRelayExport) + recreated := &corev1.Service{ + Name: frontendKey.Name, Namespace: frontendKey.Namespace, + Spec: corev1.ServiceSpec{Ports: []corev1.ServicePort{{Name: "http", Port: 8000}}}, + } + if err := controllerutil.SetControllerReference(second, recreated, apiClient.Scheme()); err != nil { + t.Fatal(err) + } + if err := apiClient.Create(ctx, recreated); err != nil { + t.Fatal(err) + } + waitForExports(t, apiClient, relay, poolRelayExport, relay.Name+"-web") + t.Log("With the controller stopped, replace the DGD at once, so that no cache sees it missing") stop() if err := apiClient.Delete(ctx, second); err != nil { @@ -499,7 +542,7 @@ func TestRelayControllerRetriesFailedExports(t *testing.T) { t.Fatal(err) } frontendExport := relay.Name + "-frontend" - waitForExports(t, apiClient, relay, serviceName(relay.Name), frontendExport) + waitForExports(t, apiClient, relay, multicluster.PoolRelayServiceName(relay.Name), frontendExport) markRelayListenersAvailable(t, apiClient, relay) t.Log("The API server refuses the next two ServiceExport creates, and a user deletes the frontend export") @@ -511,7 +554,7 @@ func TestRelayControllerRetriesFailedExports(t *testing.T) { } t.Log("Only the retries of the failed reconciles requeue the Ready Relay: the export is created again, with no other change") - waitForExports(t, apiClient, relay, serviceName(relay.Name), frontendExport) + waitForExports(t, apiClient, relay, multicluster.PoolRelayServiceName(relay.Name), frontendExport) if exportsByName(t, apiClient, relay.Namespace)[frontendExport].GetUID() == deleted.GetUID() { t.Fatal("the frontend export was not created again") } @@ -546,7 +589,7 @@ func TestRelayControllerWorksAtTheResolvedVersion(t *testing.T) { poolRelayExport := func() *unstructured.Unstructured { items := listServiceExports(t, apiClient, version, relay.Namespace) for i := range items { - if items[i].GetName() == serviceName(relay.Name) && ownedByRelay(&items[i], relay) { + if items[i].GetName() == multicluster.PoolRelayServiceName(relay.Name) && ownedByRelay(&items[i], relay) { return &items[i] } } @@ -591,12 +634,12 @@ func TestNamespaceRestrictedRelayControllerFollowsTheClusterID(t *testing.T) { if err := apiClient.Create(ctx, relay); err != nil { t.Fatal(err) } - waitForExports(t, apiClient, relay, serviceName(relay.Name), relay.Name+"-frontend") + waitForExports(t, apiClient, relay, multicluster.PoolRelayServiceName(relay.Name), relay.Name+"-frontend") t.Log("Each export's creation is an event of its own, though both events regard the Relay at one version") eventually(t, "a CreateServiceExport event for each export", func() (bool, error) { notes := strings.Join(relayEventNotes(t, apiClient, relay, "CreateServiceExport"), "\n") - return strings.Contains(notes, "Service "+serviceName(relay.Name)+" ") && + return strings.Contains(notes, "Service "+multicluster.PoolRelayServiceName(relay.Name)+" ") && strings.Contains(notes, "Service "+relay.Name+"-frontend "), nil }) markRelayListenersAvailable(t, apiClient, relay) diff --git a/deploy/operator/internal/controller/poolrelay/exports_test.go b/deploy/operator/internal/controller/poolrelay/exports_test.go index d3897adf7aff..86a37ba02718 100644 --- a/deploy/operator/internal/controller/poolrelay/exports_test.go +++ b/deploy/operator/internal/controller/poolrelay/exports_test.go @@ -9,6 +9,7 @@ import ( "context" "errors" "net/http" + "slices" "strings" "testing" "time" @@ -130,6 +131,35 @@ func ownedExport( return export } +// poolServices returns the Services that the operator creates for the pool: the +// PoolRelay Service, which relay controls, and a Service for each frontend, which +// dgd controls. It leaves out the Services named in skip. +func poolServices( + t *testing.T, + scheme *runtime.Scheme, + relay *v1alpha1.DynamoPoolRelay, + dgd *v1beta1.DynamoGraphDeployment, + skip ...string, +) []client.Object { + t.Helper() + var services []client.Object + for _, name := range multicluster.ServiceNames(dgd) { + if slices.Contains(skip, name) { + continue + } + var controller metav1.Object = dgd + if name == multicluster.PoolRelayServiceName(dgd.Name) { + controller = relay + } + service := &corev1.Service{Name: name, Namespace: dgd.Namespace} + if err := controllerutil.SetControllerReference(controller, service, scheme); err != nil { + t.Fatalf("set the controller of Service %s: %v", name, err) + } + services = append(services, service) + } + return services +} + // listServiceExports returns the ServiceExports in namespace, read at version. func listServiceExports(t *testing.T, c client.Client, version, namespace string) []unstructured.Unstructured { t.Helper() @@ -222,10 +252,22 @@ func TestReconcileExports(t *testing.T) { ownedByAnother []string // elsewhere are a user's exports in namespace other. elsewhere []string - wantOwned []string + // foreignServices are Services of the pool's names that no one controls, + // missingServices do not exist, and dcdServices are controlled by a DCD + // that the DGD controls. + foreignServices []string + missingServices []string + dcdServices []string + // otherServices are controlled by another DGD, and otherDCDServices by a + // DCD of another DGD. + otherServices []string + otherDCDServices []string + wantOwned []string // wantReasons are the reasons of the events of the first reconcile. The - // second records only its warnings again. + // second changes nothing, so it records none. wantReasons []string + // wantCondition is the reason of the Relay's Exported condition. + wantCondition string }{ { name: "a pool exports its PoolRelay and frontend Services", @@ -233,6 +275,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationKarmada, wantOwned: []string{"green-pool-relay", "green-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "each frontend component's Service is exported under its own name", @@ -240,6 +283,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationStandard, wantOwned: []string{"green-pool-relay", "green-frontend", "green-web-v2"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "a renamed frontend's export moves to its new Service", @@ -248,6 +292,7 @@ func TestReconcileExports(t *testing.T) { ownedBefore: []string{"green-pool-relay", "green-frontend"}, wantOwned: []string{"green-pool-relay", "green-web"}, wantReasons: []string{"CreateServiceExport", "DeleteServiceExport"}, + wantCondition: reasonExported, }, { name: "a pool without a frontend exports its PoolRelay Service and warns", @@ -255,6 +300,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationStandard, wantOwned: []string{"green-pool-relay"}, wantReasons: []string{"CreateServiceExport", reasonNoFrontend}, + wantCondition: reasonNoFrontend, }, { // A Service name has at most 63 characters, and the operator does not @@ -264,6 +310,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationKarmada, wantOwned: []string{"green-pool-relay", "green-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport", reasonInvalidServiceName}, + wantCondition: reasonInvalidServiceName, }, { name: "an export a user created is neither adopted nor deleted", @@ -272,6 +319,7 @@ func TestReconcileExports(t *testing.T) { foreign: []string{"green-frontend"}, wantOwned: []string{"green-pool-relay"}, wantReasons: []string{"CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "another Relay's export of a Service the pool no longer has is left alone", @@ -280,6 +328,7 @@ func TestReconcileExports(t *testing.T) { ownedByAnother: []string{"green-old"}, wantOwned: []string{"green-pool-relay", "green-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "Karmada skips a Service name it cannot import", @@ -287,6 +336,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationKarmada, wantOwned: []string{longName + "-frontend"}, wantReasons: []string{"CreateServiceExport", reasonServiceNameTooLong}, + wantCondition: reasonServiceNameTooLong, }, { name: "standard MCS exports the same long names", @@ -294,6 +344,7 @@ func TestReconcileExports(t *testing.T) { implementation: configv1alpha1.MCSImplementationStandard, wantOwned: []string{longName + "-pool-relay", longName + "-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "Karmada withdraws the Relay's export of a Service name it cannot import", @@ -302,6 +353,62 @@ func TestReconcileExports(t *testing.T) { ownedBefore: []string{longName + "-pool-relay", longName + "-frontend"}, wantOwned: []string{longName + "-frontend"}, wantReasons: []string{"DeleteServiceExport", reasonServiceNameTooLong}, + wantCondition: reasonServiceNameTooLong, + }, + { + name: "a Service that the pool does not control is not exported", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + foreignServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport", reasonServiceNotOwned}, + wantCondition: reasonServiceNotOwned, + }, + { + name: "the Relay withdraws its export of a Service that the pool no longer controls", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + ownedBefore: []string{"green-pool-relay", "green-frontend"}, + foreignServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"DeleteServiceExport", reasonServiceNotOwned}, + wantCondition: reasonServiceNotOwned, + }, + { + name: "a Service that another DGD controls is not exported", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + otherServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport", reasonServiceNotOwned}, + wantCondition: reasonServiceNotOwned, + }, + { + name: "a Service that a DCD of another DGD controls is not exported", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + otherDCDServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport", reasonServiceNotOwned}, + wantCondition: reasonServiceNotOwned, + }, + { + name: "a Service that does not exist yet is not exported", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationStandard, + missingServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay"}, + wantReasons: []string{"CreateServiceExport"}, + wantCondition: reasonExported, + }, + { + name: "a frontend Service that a DCD of the DGD controls is exported", + dgdName: "green", frontends: []string{"Frontend"}, + implementation: configv1alpha1.MCSImplementationKarmada, + dcdServices: []string{"green-frontend"}, + wantOwned: []string{"green-pool-relay", "green-frontend"}, + wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, { name: "exports of the same names in another namespace do not count", @@ -310,6 +417,7 @@ func TestReconcileExports(t *testing.T) { elsewhere: []string{"green-pool-relay", "green-frontend"}, wantOwned: []string{"green-pool-relay", "green-frontend"}, wantReasons: []string{"CreateServiceExport", "CreateServiceExport"}, + wantCondition: reasonExported, }, } { t.Run(tc.name, func(t *testing.T) { @@ -321,6 +429,28 @@ func TestReconcileExports(t *testing.T) { dgd := newExportTestDGD(tc.dgdName, tc.frontends...) relay := newExportTestRelay(dgd) objects := []client.Object{relay, dgd} + skip := slices.Concat(tc.foreignServices, tc.missingServices, tc.dcdServices, tc.otherServices, tc.otherDCDServices) + objects = append(objects, poolServices(t, scheme, relay, dgd, skip...)...) + for _, name := range tc.foreignServices { + objects = append(objects, &corev1.Service{Name: name, Namespace: dgd.Namespace}) + } + other := newExportTestDGD("other") + for _, name := range tc.otherServices { + service := &corev1.Service{Name: name, Namespace: dgd.Namespace} + g.Expect(controllerutil.SetControllerReference(other, service, scheme)).To(gomega.Succeed()) + objects = append(objects, service) + } + for _, name := range slices.Concat(tc.dcdServices, tc.otherDCDServices) { + parent := dgd + if slices.Contains(tc.otherDCDServices, name) { + parent = other + } + dcd := &v1beta1.DynamoComponentDeployment{Name: name, Namespace: dgd.Namespace, UID: uuid.NewUUID()} + service := &corev1.Service{Name: name, Namespace: dgd.Namespace} + g.Expect(controllerutil.SetControllerReference(parent, dcd, scheme)).To(gomega.Succeed()) + g.Expect(controllerutil.SetControllerReference(dcd, service, scheme)).To(gomega.Succeed()) + objects = append(objects, dcd, service) + } for _, name := range tc.ownedBefore { objects = append(objects, ownedExport(t, scheme, exportTestVersion, relay, name)) } @@ -343,15 +473,21 @@ func TestReconcileExports(t *testing.T) { first := eventReasons(drainEvents(recorder)) g.Expect(first).To(gomega.ConsistOf(tc.wantReasons)) - t.Log("Reconcile again: every export is in place, so only the warnings are recorded again") + t.Log("Reconcile again: every export is in place and the condition is unchanged, so nothing is recorded") g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) - var warnings []string - for _, reason := range first { - if !strings.HasSuffix(reason, "ServiceExport") { - warnings = append(warnings, reason) - } + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + + t.Log("Check the Relay's Exported condition") + stored := &v1alpha1.DynamoPoolRelay{} + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(relay), stored)).To(gomega.Succeed()) + condition := meta.FindStatusCondition(stored.Status.Conditions, conditionExported) + g.Expect(condition).NotTo(gomega.BeNil()) + g.Expect(condition.Reason).To(gomega.Equal(tc.wantCondition)) + wantStatus := metav1.ConditionFalse + if tc.wantCondition == reasonExported { + wantStatus = metav1.ConditionTrue } - g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf(warnings)) + g.Expect(condition.Status).To(gomega.Equal(wantStatus)) t.Log("Check which exports the Relay controls, and that other exports are untouched") var owned, others []string @@ -400,6 +536,14 @@ func TestWithdrawExports(t *testing.T) { g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("DeleteServiceExport", "DeleteServiceExport")) g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). To(gomega.ConsistOf("green-old", "green-web", "blue-frontend")) + + t.Log("The Relay's Exported condition says that the pool's exports are withdrawn") + stored := &v1alpha1.DynamoPoolRelay{} + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(relay), stored)).To(gomega.Succeed()) + condition := meta.FindStatusCondition(stored.Status.Conditions, conditionExported) + g.Expect(condition).NotTo(gomega.BeNil()) + g.Expect(condition.Status).To(gomega.Equal(metav1.ConditionFalse)) + g.Expect(condition.Reason).To(gomega.Equal(reasonWithdrawn)) } func TestRelayControllerMaintainsThePoolsExports(t *testing.T) { @@ -410,43 +554,47 @@ func TestRelayControllerMaintainsThePoolsExports(t *testing.T) { scheme := newExportTestScheme(t) first := newExportTestDGD("green", "Frontend") relay := newExportTestRelay(first) - fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, first) + poolRelayService := multicluster.PoolRelayServiceName(relay.Name) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + append([]client.Object{relay, first}, poolServices(t, scheme, relay, first, poolRelayService)...)...) r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(50), configv1alpha1.MCSImplementationKarmada, exportTestVersion) request := ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)} - serviceKey := client.ObjectKey{Namespace: relay.Namespace, Name: serviceName(relay.Name)} - reconcileRelay := func() { - _, err := r.Reconcile(ctx, request) - g.Expect(err).NotTo(gomega.HaveOccurred()) - } - exports := func() []string { - return exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace)) - } + serviceKey := client.ObjectKey{Namespace: relay.Namespace, Name: poolRelayService} t.Log("Reconcile: the Relay's Service is published, and the pool's exports are created") - reconcileRelay() + _, err := r.Reconcile(ctx, request) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(fakeClient.Get(ctx, serviceKey, &corev1.Service{})).To(gomega.Succeed()) - g.Expect(exports()).To(gomega.ConsistOf("green-pool-relay", "green-frontend")) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-pool-relay", "green-frontend")) t.Log("Delete the DGD: the Relay's Service and the pool's exports are withdrawn, and the Relay stays") g.Expect(fakeClient.Delete(ctx, first)).To(gomega.Succeed()) - reconcileRelay() + _, err = r.Reconcile(ctx, request) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(apierrors.IsNotFound(fakeClient.Get(ctx, serviceKey, &corev1.Service{}))).To(gomega.BeTrue()) - g.Expect(exports()).To(gomega.BeEmpty()) + g.Expect(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace)).To(gomega.BeEmpty()) - t.Log("Create a new DGD of the Relay's name with another frontend: the Relay publishes it, and its exports") + t.Log("Create a new DGD of the Relay's name with another frontend and its Service: the Relay publishes it, and its exports") second := newExportTestDGD("green", "Web") g.Expect(fakeClient.Create(ctx, second)).To(gomega.Succeed()) - reconcileRelay() + for _, service := range poolServices(t, scheme, relay, second, poolRelayService) { + g.Expect(fakeClient.Create(ctx, service)).To(gomega.Succeed()) + } + _, err = r.Reconcile(ctx, request) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(fakeClient.Get(ctx, serviceKey, &corev1.Service{})).To(gomega.Succeed()) - g.Expect(exports()).To(gomega.ConsistOf("green-pool-relay", "green-web")) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-pool-relay", "green-web")) t.Log("Leave the DGD without workers: the Relay withdraws, and the pool's exports with it") g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(second), second)).To(gomega.Succeed()) second.Spec.Components = second.Spec.Components[1:] g.Expect(fakeClient.Update(ctx, second)).To(gomega.Succeed()) - reconcileRelay() + _, err = r.Reconcile(ctx, request) + g.Expect(err).NotTo(gomega.HaveOccurred()) g.Expect(apierrors.IsNotFound(fakeClient.Get(ctx, serviceKey, &corev1.Service{}))).To(gomega.BeTrue()) - g.Expect(exports()).To(gomega.BeEmpty()) + g.Expect(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace)).To(gomega.BeEmpty()) } func TestPoolExportsDoNotWaitForTheRelaysDeployment(t *testing.T) { @@ -465,7 +613,7 @@ func TestPoolExportsDoNotWaitForTheRelaysDeployment(t *testing.T) { } return c.Create(ctx, obj, opts...) }, - }, relay, dgd) + }, append([]client.Object{relay, dgd}, poolServices(t, scheme, relay, dgd, multicluster.PoolRelayServiceName(relay.Name))...)...) r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(20), configv1alpha1.MCSImplementationKarmada, exportTestVersion) t.Log("Reconcile: the Deployment's failure is returned, and the pool's exports are created anyway") @@ -490,7 +638,7 @@ func TestRelayControllerReturnsAFailedExportWrite(t *testing.T) { } return c.Create(ctx, obj, opts...) }, - }, relay, dgd) + }, append([]client.Object{relay, dgd}, poolServices(t, scheme, relay, dgd, multicluster.PoolRelayServiceName(relay.Name))...)...) recorder := events.NewFakeRecorder(20) r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) @@ -498,7 +646,7 @@ func TestRelayControllerReturnsAFailedExportWrite(t *testing.T) { result, err := r.Reconcile(ctx, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(relay)}) g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) g.Expect(result).To(gomega.Equal(ctrl.Result{})) - key := client.ObjectKey{Namespace: relay.Namespace, Name: serviceName(relay.Name)} + key := client.ObjectKey{Namespace: relay.Namespace, Name: multicluster.PoolRelayServiceName(relay.Name)} g.Expect(fakeClient.Get(ctx, key, &appsv1.Deployment{})).To(gomega.Succeed()) g.Expect(countReasons(drainEvents(recorder), reasonExportFailed)).To(gomega.Equal(1)) } @@ -511,7 +659,7 @@ func TestOwnershipConflictWithdrawsThePoolsExports(t *testing.T) { scheme := newExportTestScheme(t) dgd := newExportTestDGD("green", "Frontend") relay := newExportTestRelay(dgd) - foreign := &corev1.Service{Name: serviceName(relay.Name), Namespace: relay.Namespace} + foreign := &corev1.Service{Name: multicluster.PoolRelayServiceName(relay.Name), Namespace: relay.Namespace} fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, dgd, foreign, ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), ownedExport(t, scheme, exportTestVersion, relay, "green-frontend")) @@ -531,8 +679,9 @@ func TestReconcileExportsUsesTheResolvedVersion(t *testing.T) { scheme := newExportTestScheme(t) dgd := newExportTestDGD("green", "Web") relay := newExportTestRelay(dgd) - fakeClient := newExportTestClient(scheme, version, interceptor.Funcs{}, - dgd, relay, ownedExport(t, scheme, version, relay, "green-frontend")) + fakeClient := newExportTestClient(scheme, version, interceptor.Funcs{}, append([]client.Object{ + dgd, relay, ownedExport(t, scheme, version, relay, "green-frontend"), + }, poolServices(t, scheme, relay, dgd)...)...) t.Log("Reconcile with the gates resolved to that version: exports are listed, created and withdrawn at it") recorder := events.NewFakeRecorder(10) @@ -558,7 +707,7 @@ func TestReconcileExportsToleratesAStaleCache(t *testing.T) { return apierrors.NewAlreadyExists( schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, obj.GetName()) }, - }, dgd, relay) + }, append([]client.Object{dgd, relay}, poolServices(t, scheme, relay, dgd)...)...) t.Log("Reconcile: AlreadyExists is no error, and nothing is announced") recorder := events.NewFakeRecorder(10) @@ -575,7 +724,10 @@ func TestPoolExportsRetryFailedWrites(t *testing.T) { failOnce func(failed *bool) interceptor.Funcs // withdraw withdraws the exports instead of reconciling them. withdraw bool - want []string + // wantAfterFailure are the exports after the failed write: the failure + // does not hold back the other export's write. + wantAfterFailure []string + want []string }{ { name: "a failed create", @@ -590,7 +742,8 @@ func TestPoolExportsRetryFailedWrites(t *testing.T) { }, } }, - want: []string{"green-pool-relay", "green-frontend"}, + wantAfterFailure: []string{"green-frontend"}, + want: []string{"green-pool-relay", "green-frontend"}, }, { name: "a failed delete", @@ -605,7 +758,8 @@ func TestPoolExportsRetryFailedWrites(t *testing.T) { }, } }, - withdraw: true, + withdraw: true, + wantAfterFailure: []string{"green-frontend"}, }, } { t.Run(tc.name, func(t *testing.T) { @@ -616,7 +770,7 @@ func TestPoolExportsRetryFailedWrites(t *testing.T) { scheme := newExportTestScheme(t) dgd := newExportTestDGD("green", "Frontend") relay := newExportTestRelay(dgd) - objects := []client.Object{relay, dgd} + objects := append([]client.Object{relay, dgd}, poolServices(t, scheme, relay, dgd)...) if tc.withdraw { objects = append(objects, ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), @@ -626,19 +780,25 @@ func TestPoolExportsRetryFailedWrites(t *testing.T) { fakeClient := newExportTestClient(scheme, exportTestVersion, tc.failOnce(&failed), objects...) recorder := events.NewFakeRecorder(10) r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) - step := func() error { - if tc.withdraw { - return r.withdrawExports(ctx, relay) - } - return r.reconcileExports(ctx, relay, dgd) - } t.Log("The failure is returned, for controller-runtime to retry with backoff, and is announced") - g.Expect(step()).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + var err error + if tc.withdraw { + err = r.withdrawExports(ctx, relay) + } else { + err = r.reconcileExports(ctx, relay, dgd) + } + g.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) g.Expect(drainEvents(recorder)).To(gomega.ContainElement(gomega.HavePrefix(corev1.EventTypeWarning + " " + reasonExportFailed))) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))).To(gomega.ConsistOf(tc.wantAfterFailure)) t.Log("The retry runs once the API server answers: the exports converge with no other change") - g.Expect(step()).To(gomega.Succeed()) + if tc.withdraw { + err = r.withdrawExports(ctx, relay) + } else { + err = r.reconcileExports(ctx, relay, dgd) + } + g.Expect(err).To(gomega.Succeed()) g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))).To(gomega.ConsistOf(tc.want)) }) } @@ -693,8 +853,9 @@ func TestPoolExportsAreRepaired(t *testing.T) { changed.SetAnnotations(map[string]string{"example.com/note": "kept"}) changed.SetLabels(map[string]string{commonconsts.KubeLabelDynamoGraphDeploymentName: dgd.Name, "team": "router"}) tc.drift(changed) - fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, - dgd, relay, intact.DeepCopy(), changed.DeepCopy()) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, append([]client.Object{ + dgd, relay, intact.DeepCopy(), changed.DeepCopy(), + }, poolServices(t, scheme, relay, dgd)...)...) t.Log("Reconcile: the export is repaired, and the repair is recorded") recorder := events.NewFakeRecorder(10) @@ -733,8 +894,9 @@ func TestPoolExportsBeingDeletedAreLeftAlone(t *testing.T) { terminating.SetLabels(nil) terminating.SetFinalizers([]string{"example.com/hold"}) terminating.SetDeletionTimestamp(&metav1.Time{Time: time.Now()}) - fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, dgd, relay, - ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), terminating.DeepCopy()) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, append([]client.Object{ + dgd, relay, ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), terminating.DeepCopy(), + }, poolServices(t, scheme, relay, dgd)...)...) t.Log("Reconcile: while its deletion is pending, the export is neither repaired nor created again") recorder := events.NewFakeRecorder(10) @@ -765,8 +927,9 @@ func staleCacheSetUp(t *testing.T) ( frontendExport.SetUID("export-frontend") frontendExport.SetLabels(nil) cache := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relayExport.DeepCopy(), frontendExport.DeepCopy()) - apiServer := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, - relay, dgd, relayExport.DeepCopy(), frontendExport.DeepCopy()) + apiServer := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, append([]client.Object{ + relay, dgd, relayExport.DeepCopy(), frontendExport.DeepCopy(), + }, poolServices(t, scheme, relay, dgd)...)...) recorder := events.NewFakeRecorder(10) r := newExportTestReconciler(apiServer, recorder, configv1alpha1.MCSImplementationKarmada, exportTestVersion) r.exports.cache = cache @@ -977,14 +1140,74 @@ func TestPoolsFollowTheClusterIDProperty(t *testing.T) { blue := newExportTestDGD("blue", "Frontend") greenRelay, blueRelay := newExportTestRelay(green), newExportTestRelay(blue) property := newClusterIDProperty("east") - fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, green, blue, greenRelay, blueRelay, property) + objects := []client.Object{green, blue, greenRelay, blueRelay, property} + objects = append(objects, poolServices(t, scheme, greenRelay, green)...) + objects = append(objects, poolServices(t, scheme, blueRelay, blue)...) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, objects...) recorder := events.NewFakeRecorder(20) r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) r.exports.clusterPropertyWatched = true + pools := []ctrl.Request{ + {NamespacedName: client.ObjectKeyFromObject(greenRelay)}, + {NamespacedName: client.ObjectKeyFromObject(blueRelay)}, + } - // mismatches reconciles the exports of the Relays of requests, and returns how - // many of them report a mismatch. - mismatches := func(requests []ctrl.Request) int { + for _, step := range []struct { + description string + // relayClusterID, when set, is the clusterId that users give both Relays, + // whose own watch requeues them. + relayClusterID string + // propertyValue, when set, is the property's new value. + propertyValue string + // deleteProperty deletes the property, and createProperty, when set, + // creates it again with that value. + deleteProperty bool + createProperty string + // wantMismatches is how many Relays record a mismatch. + wantMismatches int + }{ + {description: "Reconcile both pools: their IDs match the property"}, + { + description: "The cluster is renamed west: users correct both Relays first, and their watch requeues them while the cache still says east", + relayClusterID: "west", wantMismatches: 2, + }, + { + description: "The cache sees the property's new value: its event requeues both Relays, which no longer report a mismatch", + propertyValue: "west", + }, + {description: "The value changes again: both Relays report the mismatch", propertyValue: "south", wantMismatches: 2}, + {description: "The property is deleted: its event requeues both Relays, which skip the check", deleteProperty: true}, + { + description: "The property is created again with another value: its event requeues both Relays, which report the mismatch", + createProperty: "north", wantMismatches: 2, + }, + } { + t.Log(step.description) + requests := pools + switch { + case step.relayClusterID != "": + for _, relay := range []*v1alpha1.DynamoPoolRelay{greenRelay, blueRelay} { + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(relay), relay)).To(gomega.Succeed()) + relay.Spec.ClusterID = step.relayClusterID + g.Expect(fakeClient.Update(ctx, relay)).To(gomega.Succeed()) + } + case step.propertyValue != "": + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(property), property)).To(gomega.Succeed()) + before := property.DeepCopy() + property.Object["spec"] = map[string]any{"value": step.propertyValue} + g.Expect(fakeClient.Update(ctx, property)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: before, ObjectNew: property})).To(gomega.BeTrue()) + requests = r.mapClusterPropertyToRelays(ctx, property) + case step.deleteProperty: + g.Expect(fakeClient.Delete(ctx, property)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Delete(event.DeleteEvent{Object: property})).To(gomega.BeTrue()) + requests = r.mapClusterPropertyToRelays(ctx, property) + case step.createProperty != "": + created := newClusterIDProperty(step.createProperty) + g.Expect(fakeClient.Create(ctx, created)).To(gomega.Succeed()) + g.Expect(clusterIDPropertyEvents.Create(event.CreateEvent{Object: created})).To(gomega.BeTrue()) + requests = r.mapClusterPropertyToRelays(ctx, created) + } for _, request := range requests { relay := &v1alpha1.DynamoPoolRelay{} g.Expect(fakeClient.Get(ctx, request.NamespacedName, relay)).To(gomega.Succeed()) @@ -992,52 +1215,185 @@ func TestPoolsFollowTheClusterIDProperty(t *testing.T) { g.Expect(fakeClient.Get(ctx, request.NamespacedName, dgd)).To(gomega.Succeed()) g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) } - return countReasons(drainEvents(recorder), reasonClusterIDMismatch) - } - pools := []ctrl.Request{ - {NamespacedName: client.ObjectKeyFromObject(greenRelay)}, - {NamespacedName: client.ObjectKeyFromObject(blueRelay)}, + g.Expect(countReasons(drainEvents(recorder), reasonClusterIDMismatch)).To(gomega.Equal(step.wantMismatches)) } - g.Expect(mismatches(pools)).To(gomega.BeZero()) +} - t.Log("The cluster is renamed west: users correct both Relays first, and their watch requeues them while the cache still says east") - for _, relay := range []*v1alpha1.DynamoPoolRelay{greenRelay, blueRelay} { - g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(relay), relay)).To(gomega.Succeed()) - relay.Spec.ClusterID = "west" - g.Expect(fakeClient.Update(ctx, relay)).To(gomega.Succeed()) - } - g.Expect(mismatches(pools)).To(gomega.Equal(2)) - - // change applies update to the property in the cache, and returns the Relays - // that the resulting event requeues. - change := func(update func(property *unstructured.Unstructured)) []ctrl.Request { - g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(property), property)).To(gomega.Succeed()) - before := property.DeepCopy() - update(property) - g.Expect(fakeClient.Update(ctx, property)).To(gomega.Succeed()) - g.Expect(clusterIDPropertyEvents.Update(event.UpdateEvent{ObjectOld: before, ObjectNew: property})).To(gomega.BeTrue()) - return r.mapClusterPropertyToRelays(ctx, property) - } - setValue := func(value string) func(*unstructured.Unstructured) { - return func(property *unstructured.Unstructured) { property.Object["spec"] = map[string]any{"value": value} } +func TestPoolExportsConvergePastAFailedWrite(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool whose frontend was renamed from Frontend to Web, in a cluster whose ID differs from the " + + "Relay's, while a quota refuses every new export") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Web") + relay := newExportTestRelay(dgd) + quota := apierrors.NewForbidden(schema.GroupResource{Group: multicluster.ServiceExportGroupKind.Group, Resource: "serviceexports"}, + "green-web", errors.New("exceeded quota")) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if obj.GetObjectKind().GroupVersionKind().GroupKind() == multicluster.ServiceExportGroupKind { + return quota + } + return c.Create(ctx, obj, opts...) + }, + }, append([]client.Object{ + relay, dgd, newClusterIDProperty("west"), + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend"), + }, poolServices(t, scheme, relay, dgd)...)...) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + r.exports.clusterPropertyWatched = true + + t.Log("Reconcile: the refused create is returned, but the stale export is withdrawn, which frees the quota, " + + "and the mismatch is still reported") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.MatchError(gomega.ContainSubstring("exceeded quota"))) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))).To(gomega.ConsistOf("green-pool-relay")) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("DeleteServiceExport", reasonExportFailed, reasonClusterIDMismatch)) + condition := meta.FindStatusCondition(relay.Status.Conditions, conditionExported) + g.Expect(condition).NotTo(gomega.BeNil()) + g.Expect(condition.Reason).To(gomega.Equal(reasonExportFailed)) + g.Expect(condition.Message).To(gomega.And(gomega.ContainSubstring("exceeded quota"), gomega.ContainSubstring(`clusterId "east"`))) +} + +func TestPoolExportProblemsAreReportedPastAFailedWithdrawal(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool whose frontend was renamed from Frontend to Web, in a cluster whose ID differs from the " + + "Relay's, while the API server refuses deletes") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Web") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return apierrors.NewServiceUnavailable("the API server is restarting") + }, + }, append([]client.Object{ + relay, dgd, newClusterIDProperty("west"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend"), + }, poolServices(t, scheme, relay, dgd)...)...) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + r.exports.clusterPropertyWatched = true + + t.Log("Reconcile: the refused delete is returned, the new exports are created, and the mismatch is still reported") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-pool-relay", "green-web", "green-frontend")) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf( + "CreateServiceExport", "CreateServiceExport", reasonExportFailed, reasonClusterIDMismatch)) +} + +func TestPoolExportsSurviveAFailedServiceRead(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool with both exports while reading its Services fails") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green", "Frontend") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*corev1.Service); ok { + return apierrors.NewServiceUnavailable("the API server is restarting") + } + return c.Get(ctx, key, obj, opts...) + }, + }, append([]client.Object{ + relay, dgd, + ownedExport(t, scheme, exportTestVersion, relay, "green-pool-relay"), + ownedExport(t, scheme, exportTestVersion, relay, "green-frontend"), + }, poolServices(t, scheme, relay, dgd)...)...) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(10), configv1alpha1.MCSImplementationStandard, exportTestVersion) + + t.Log("Reconcile: the failure is returned, and the exports stay until the Services can be read again") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.MatchError(gomega.ContainSubstring("restarting"))) + g.Expect(exportNames(listServiceExports(t, fakeClient, exportTestVersion, relay.Namespace))). + To(gomega.ConsistOf("green-pool-relay", "green-frontend")) +} + +func TestPoolExportWarningsFollowTheExportedCondition(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() + + t.Log("Create a pool without a frontend") + scheme := newExportTestScheme(t) + dgd := newExportTestDGD("green") + relay := newExportTestRelay(dgd) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + append([]client.Object{relay, dgd}, poolServices(t, scheme, relay, dgd)...)...) + recorder := events.NewFakeRecorder(10) + r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) + + t.Log("Reconcile: the condition reports the problem, and its Warning is recorded once") + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("CreateServiceExport", reasonNoFrontend)) + + t.Log("A DGD status write changes only the DGD's resourceVersion, so the condition is unchanged and nothing is recorded") + g.Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(dgd), dgd)).To(gomega.Succeed()) + dgd.Status.State = "pending" + g.Expect(fakeClient.Status().Update(ctx, dgd)).To(gomega.Succeed()) + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(drainEvents(recorder)).To(gomega.BeEmpty()) + + t.Log("A frontend is added: the condition turns True, and only the new export is recorded") + dgd.Spec.Components = append(dgd.Spec.Components, v1beta1.DynamoComponentDeploymentSharedSpec{ + ComponentName: "Frontend", ComponentType: v1beta1.ComponentTypeFrontend, + }) + g.Expect(fakeClient.Update(ctx, dgd)).To(gomega.Succeed()) + for _, service := range poolServices(t, scheme, relay, dgd, multicluster.PoolRelayServiceName(dgd.Name)) { + g.Expect(fakeClient.Create(ctx, service)).To(gomega.Succeed()) } + g.Expect(r.reconcileExports(ctx, relay, dgd)).To(gomega.Succeed()) + g.Expect(eventReasons(drainEvents(recorder))).To(gomega.ConsistOf("CreateServiceExport")) + condition := meta.FindStatusCondition(relay.Status.Conditions, conditionExported) + g.Expect(condition).NotTo(gomega.BeNil()) + g.Expect(condition.Status).To(gomega.Equal(metav1.ConditionTrue)) + g.Expect(condition.Message).To(gomega.Equal("Exported Services green-pool-relay, green-frontend through MCS implementation mcs")) +} - t.Log("The cache sees the property's new value: its event requeues both Relays, which no longer report a mismatch") - g.Expect(mismatches(change(setValue("west")))).To(gomega.BeZero()) +func TestMapServiceToRelays(t *testing.T) { + g := gomega.NewGomegaWithT(t) + ctx := context.Background() - t.Log("The value changes again: both Relays report the mismatch") - g.Expect(mismatches(change(setValue("south")))).To(gomega.Equal(2)) + t.Log("Create two pools in namespace inference, one with a frontend named Frontend") + scheme := newExportTestScheme(t) + green := newExportTestDGD("green", "Frontend") + blue := newExportTestDGD("blue") + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, + green, blue, newExportTestRelay(green), newExportTestRelay(blue)) + r := newExportTestReconciler(fakeClient, events.NewFakeRecorder(1), configv1alpha1.MCSImplementationStandard, exportTestVersion) - t.Log("The property is deleted: its event requeues both Relays, which skip the check") - g.Expect(fakeClient.Delete(ctx, property)).To(gomega.Succeed()) - g.Expect(clusterIDPropertyEvents.Delete(event.DeleteEvent{Object: property})).To(gomega.BeTrue()) - g.Expect(mismatches(r.mapClusterPropertyToRelays(ctx, property))).To(gomega.BeZero()) + t.Log("A Service requeues the Relays whose DGD exports a Service of its name") + g.Expect(r.mapServiceToRelays(ctx, &corev1.Service{Name: "green-frontend", Namespace: "inference"})). + To(gomega.ConsistOf(ctrl.Request{Namespace: "inference", Name: "green"})) + g.Expect(r.mapServiceToRelays(ctx, &corev1.Service{Name: "blue-pool-relay", Namespace: "inference"})). + To(gomega.ConsistOf(ctrl.Request{Namespace: "inference", Name: "blue"})) + g.Expect(r.mapServiceToRelays(ctx, &corev1.Service{Name: "blue-frontend", Namespace: "inference"})).To(gomega.BeEmpty()) + g.Expect(r.mapServiceToRelays(ctx, &corev1.Service{Name: "green-frontend", Namespace: "other"})).To(gomega.BeEmpty()) + + t.Log("A Service's creation and deletion count, and an update only when it changes the Service's controller") + scheme = newExportTestScheme(t) + service := &corev1.Service{Name: "green-frontend", Namespace: "inference"} + g.Expect(serviceControllerChanged.Create(event.CreateEvent{Object: service})).To(gomega.BeTrue()) + g.Expect(serviceControllerChanged.Delete(event.DeleteEvent{Object: service})).To(gomega.BeTrue()) + relabeled := service.DeepCopy() + relabeled.Labels = map[string]string{"team": "router"} + g.Expect(serviceControllerChanged.Update(event.UpdateEvent{ObjectOld: service, ObjectNew: relabeled})).To(gomega.BeFalse()) + controlled := service.DeepCopy() + g.Expect(controllerutil.SetControllerReference(green, controlled, scheme)).To(gomega.Succeed()) + g.Expect(serviceControllerChanged.Update(event.UpdateEvent{ObjectOld: service, ObjectNew: controlled})).To(gomega.BeTrue()) + g.Expect(serviceControllerChanged.Update(event.UpdateEvent{ObjectOld: controlled, ObjectNew: service})).To(gomega.BeTrue()) +} - t.Log("The property is created again with another value: its event requeues both Relays, which report the mismatch") - created := newClusterIDProperty("north") - g.Expect(fakeClient.Create(ctx, created)).To(gomega.Succeed()) - g.Expect(clusterIDPropertyEvents.Create(event.CreateEvent{Object: created})).To(gomega.BeTrue()) - g.Expect(mismatches(r.mapClusterPropertyToRelays(ctx, created))).To(gomega.Equal(2)) +func TestTruncateKeepsWholeCharacters(t *testing.T) { + g := gomega.NewGomegaWithT(t) + g.Expect(truncate("short", 10)).To(gomega.Equal("short")) + g.Expect(truncate("ab€cd", 4)).To(gomega.Equal("ab")) + g.Expect(truncate("ab€cd", 5)).To(gomega.Equal("ab€")) + g.Expect(truncate(strings.Repeat("x", 2000), maxEventNote)).To(gomega.HaveLen(maxEventNote)) } func TestMapServiceExportToRelays(t *testing.T) { @@ -1304,8 +1660,9 @@ func TestPoolExportEventsNameTheirObjects(t *testing.T) { relay := newExportTestRelay(dgd) unlabeled := ownedExport(t, scheme, exportTestVersion, relay, "green-frontend") unlabeled.SetLabels(nil) - fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, relay, dgd, unlabeled, - ownedExport(t, scheme, exportTestVersion, relay, "green-old"), newClusterIDProperty("west")) + fakeClient := newExportTestClient(scheme, exportTestVersion, interceptor.Funcs{}, append([]client.Object{ + relay, dgd, unlabeled, ownedExport(t, scheme, exportTestVersion, relay, "green-old"), newClusterIDProperty("west"), + }, poolServices(t, scheme, relay, dgd)...)...) recorder := &relatedRecorder{} r := newExportTestReconciler(fakeClient, recorder, configv1alpha1.MCSImplementationStandard, exportTestVersion) r.exports.clusterPropertyWatched = true diff --git a/deploy/operator/internal/multicluster/serviceexport.go b/deploy/operator/internal/multicluster/serviceexport.go index 582a6fe89676..e422412dc14e 100644 --- a/deploy/operator/internal/multicluster/serviceexport.go +++ b/deploy/operator/internal/multicluster/serviceexport.go @@ -235,8 +235,9 @@ func ClusterPropertyValue(property *unstructured.Unstructured) string { // ReadClusterIDProperty returns the About API's ClusterProperty named // ClusterIDProperty, whose value is the cluster's ID in its ClusterSet, or nil // when the cluster has no such property or does not serve the About API. Any -// other failure, such as a missing permission, is an error. reader should be the -// manager's cache, which a watch of ClusterProperties fills. +// other failure is an error. reader should be the manager's cache, which a watch +// of ClusterProperties fills; without permission to list and watch them, that +// watch never syncs, and the controller does not start. func ReadClusterIDProperty(ctx context.Context, reader client.Reader) (*unstructured.Unstructured, error) { property := NewClusterProperty() err := reader.Get(ctx, client.ObjectKey{Name: ClusterIDProperty}, property) diff --git a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md index 65e2ec8df457..4fa7872a916f 100644 --- a/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md +++ b/docs/fern/pages/kubernetes/kv-aware-routing/pool-relay.md @@ -394,6 +394,11 @@ pool. It finds the frontends by a Service name suffix, `-frontend` by default, w frontend component named `Frontend`. The operator does not know that suffix, so a frontend component with another name is exported but not found until the suffix matches it. +The operator exports a Service only while it exists and the pool controls it: the relay controls the +PoolRelay Service, and the DGD, or one of the DGD's `DynamoComponentDeployment`s, controls each +frontend's Service. A Service that does not exist yet is exported once it does, and a Service of such +a name that someone else created is not exported. + The `DynamoPoolRelay` owns these exports, so deleting it deletes them. They follow the relay's own Service: the operator withdraws them whenever it withdraws that Service, for example when the DGD is deleted or has no workers, as the relay's `Ready` condition reports, and exports a DGD created again @@ -402,8 +407,9 @@ While the relay is an export's controller, the operator restores the export's DG controller reference if someone changes them. The operator leaves alone any `ServiceExport` that the relay does not control: one that a user created, or one whose controller reference someone removed or replaced. Such an export stays when the pool is withdrawn, and is not deleted with the relay; once -it is deleted, the operator creates its own. Without a configured implementation, nothing is -exported. +it is deleted, the operator creates its own. Without a configured implementation, the operator +creates no exports. Clearing the implementation leaves the exports that it created in place, no longer +maintained, until their `DynamoPoolRelay`s are deleted. Set `clusterId` to the name that the MCS implementation gives this cluster, because the Global Discovery Service matches the pool's registration to its imports by that name: with standard MCS, @@ -411,16 +417,21 @@ the cluster's ID in its ClusterSet, which the About API's `ClusterProperty` `cluster.clusterset.k8s.io` holds; with Karmada, this cluster's member cluster name. The MCS implementation reports on each export in the export's status, with the conditions `Valid`, -`Ready` and `Conflict`; Karmada writes none. The operator records the problems that only it can see -as Warning events on the `DynamoPoolRelay`, which `kubectl describe dynamopoolrelay ` shows: +`Ready` and `Conflict`; Karmada writes none. The operator reports on the pool's exports in the +relay's `Exported` condition: `True` once the Services that the pool can export are exported, +`False` with reason `Withdrawn` while the relay is withdrawn, and otherwise `False` with the reason of +the pool's first problem below; its message lists every problem. When the condition changes, the +operator also records a Warning event for each problem. `kubectl describe dynamopoolrelay ` +shows both. | Reason | Meaning | | --- | --- | +| `ServiceNotOwned` | A Service of a name that the pool needs exists, but neither the relay nor the DGD, or one of its `DynamoComponentDeployment`s, controls it, so it is not exported. | | `InvalidServiceName` | No Service can have a name that the pool needs, so that Service is not exported. For example, a frontend component's Service name, `-`, is longer than 63 characters. | | `ServiceNameTooLong` | With `karmada`, a Service name is longer than 55 characters, which Karmada cannot import, so that Service is not exported. | | `NoFrontend` | The DGD has no frontend component, so the Global Router has nowhere to send the pool's requests. | | `ClusterIDMismatch` | With `mcs`, the relay's `clusterId` differs from the cluster's ID in its ClusterSet. The Global Discovery Service lists the pool only when they match. The operator watches the property, so creating, changing or deleting it checks every pool again. It skips the check when the cluster has no such property, or did not serve the About API when the operator started. | -| `ExportFailed` | Listing, creating, repairing or deleting an export failed. The operator retries with backoff. | +| `ExportFailed` | Listing, creating, repairing or deleting an export, reading one of the pool's Services, or reading the cluster's ID in its ClusterSet failed. The operator retries with backoff. | ## Optional mTLS Sidecar diff --git a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md index f2de2b44feba..0d9b28bfc86a 100644 --- a/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md +++ b/docs/fern/pages/reference/kubernetes-api/additional-resources/api-reference-k8s.md @@ -811,7 +811,7 @@ _Appears in:_ | `observedGeneration` _integer_ | ObservedGeneration is the Relay resource generation last reconciled by the operator. | | Optional: \{\}
| | `endpoint` _string_ | Endpoint is the cluster-local `-pool-relay..svc:5561` address.
It is empty after withdrawal or a failed reconciliation. | | Optional: \{\}
| | `availableReplicas` _integer_ | AvailableReplicas is the number of Relay Pods the Deployment reports available. | | Optional: \{\}
| -| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#condition-v1-meta) array_ | Conditions includes Ready for listener availability, not model or KV readiness.
Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound,
DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers,
InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict. | | Optional: \{\}
| +| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#condition-v1-meta) array_ | Conditions includes Ready for listener availability, not model or KV readiness.
Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound,
DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers,
InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict.
With multi-cluster export configured, Conditions also includes Exported: True
with reason Exported once the pool's Services are exported, or False with
reason Withdrawn, ServiceNotOwned, InvalidServiceName, ServiceNameTooLong,
NoFrontend, ClusterIDMismatch, or ExportFailed. | | Optional: \{\}
| #### EPPConfig diff --git a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx index 251ba19dcd0c..eba2890650cd 100644 --- a/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx +++ b/docs/fern/pages/reference/kubernetes-api/full-api-reference.mdx @@ -1378,7 +1378,7 @@ AvailableReplicas is the number of Relay Pods the Deployment reports available. -Conditions includes Ready for listener availability, not model or KV readiness. Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound, DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers, InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict. +Conditions includes Ready for listener availability, not model or KV readiness. Its reason is ListenerAvailable, DeploymentProgressing, DGDNotFound, DGDTerminating, UnsupportedDiscovery, UnsupportedSourceScope, NoWorkers, InvalidConfiguration, ReconcileFailed, or ResourceOwnershipConflict. With multi-cluster export configured, Conditions also includes Exported: True with reason Exported once the pool's Services are exported, or False with reason Withdrawn, ServiceNotOwned, InvalidServiceName, ServiceNameTooLong, NoFrontend, ClusterIDMismatch, or ExportFailed. See [Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#condition-v1-meta). **Validation:** Optional: \{\}