[atchernych]

Overview

Adds the most basic end-to-end test of multi-cluster Global Discovery, run after every merge. Two vClusters stand in for two clusters: the hub runs the Global Discovery Service, and the member runs a pool whose PoolRelay registers with it across the cluster boundary. The test checks that the hub lists the member's pool and drops it once the relay is gone.

hub vCluster                         member vCluster
Global Discovery Service  <--------  DynamoPoolRelay (operator-managed)
  --pool-discovery registrations       POST /pools on every heartbeat
  --registration-storage memory      DGD: Frontend + mocker worker

The job is non-blocking while the multi-cluster pieces land, and the test may fail until they do.

Key Changes

• tests/deploy/global_discovery/test_global_discovery.py:
  • The relay reaches the service through the hub Service's ClusterIP. vCluster syncs pods and Services to one host cluster, so an address from one vCluster is routable from a pod of the other.
  • GET /pools on the hub must list the pool under the member's cluster ID, with the model, region and frontend URL the relay registered, and say the list is complete. Heartbeats must keep it listed for three leases, and it must leave about one lease after the relay is deleted.
  • On failure the test saves both clusters' state and logs.
• tests/deploy/global_discovery/cluster.py: kubectl bound to one cluster's kubeconfig, retried with vcluster_utils.retry_vcluster_api, and HTTP through a port-forward that reopens. The dgd_utils helpers serve only the cluster that KUBECONFIG selects.
• .github/workflows/post-merge-ci.yml:
  • global-discovery-service-build builds container/Dockerfile.global-discovery-service, which now takes DOCKER_PROXY as the operator's Dockerfile does.
  • deploy-test-global-discovery creates the hub and member vClusters, each with the operator, in a host namespace of their own, and runs the test. setup-dynamo-operator and connect-vcluster always tunnel to 127.0.0.1:8443 and reuse a tunnel they find there, so the job closes the hub's tunnel before it creates the member and connects to the hub again on 8444.
  • deploy-cleanup-global-discovery deletes both vClusters and the namespace.
  • The build and the test have continue-on-error, deploy-status-check does not wait for them, and the Slack notice leaves out jobs named Global Discovery.
• Temporary: .github/workflows/pr.yaml carries copies of the three jobs, so the test runs on this PR. They come out before merge.

Product gaps

The test works around these. Each needs its own fix.

• The operator does not pass the PoolRelay's Global Discovery settings. DynamoPoolRelay has no fields for --global-discovery-url, --routing-pool, --routing-pool-region or --routing-pool-frontends (feat(pool-relay): register the routing pool with Global Discovery #18077), and the controller sets none of them. The test sets the matching DYN_RELAY_* variables in the relay's pod template.
• The relay does not log a registration it holds back. lib/llm/src/pool_relay/routing_pool.rs records why it registers nothing, for example while the DGD's serving topology has no model, only in routing_pool_unregistered_reason. That field is served by the relay's request-plane health endpoint, not the system port's /health, so a pool that never registers leaves no trace in the relay's log.
• The PoolRelay controller refuses DGDs that use envFrom. Unless a frontend or worker container also sets DYN_DISCOVERY_BACKEND and DYN_KUBE_DISCOVERY_MODE, the relay is withdrawn as UnsupportedDiscovery. The examples load hf-token-secret with envFrom (for example examples/backends/vllm/deploy/agg.yaml), so a relay for one of them is refused. The test passes the token through a secretKeyRef.

Validation

• Not yet run on a cluster: the pr.yaml jobs on this PR are its first run.
• pre-commit passes. pytest collects the test, and the marker report finds no missing category for it. The DGD and DynamoPoolRelay manifests validate against the operator's CRD schemas, and the kubectl retry and port-forward paths pass against a stub kubectl.
• An adversarial review found no blockers, and its findings are fixed.

Issues and Ongoing Work

• refs DEP-1275. Part of DEP: Pluggable Pool Discovery for the Global Router #17600.
• Later cases, listed in the test's README: DynamoDB registration storage (feat(global-router): add a DynamoDB registration store to the Global Discovery Service #17691), registrations shared between instances (feat(global-router): share registrations between discovery instances #17637), MCS mode with the operator's ServiceExports (feat(operator): export Global Router pools through the Multi-Cluster Services API #17412) and an MCS implementation that runs in a vCluster (feat(global-router): scaffold the global-discovery-mcs module and its CI #17629 to test(global-router): add the kind end-to-end test for global-discovery-mcs #17636), and a worker rolling update (fix(operator): let a PoolRelay watch both worker revisions during a rolling update #17971).
• The kind-based MCS suites in test(global-router): add the kind end-to-end test for global-discovery-mcs #17636 and test(global-router): run shared MCS discovery cases with Cilium #17692 use stand-in router and relay pods; this test runs the real operator and PoolRelay.
• Before merge, remove the temporary pr.yaml jobs.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[devin-ai-integration]

❌ Dynamo PR CI failed — run 41948641293 (attempt 1) on 9cdec8f76b

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Other	Jobs
Global Discovery Deploy Test	❌ 1

Failure details

1 job failed: the Global Discovery Deploy Test could not create its second (member) vCluster because vCluster refuses to create more than one virtual cluster in the same host namespace.

❌ Global Discovery Deploy Test: vcluster create fatal: already a virtual cluster in namespace

Job: Global Discovery Deploy Test · Failed step: Create the member vCluster · Logs: gh run view --job 126994131613 -R ai-dynamo/dynamo --log-failed

Checking for existing vCluster ci-41948641293-gd-member in gh-id-41948641293-global-discovery-dt
No existing vCluster named ci-41948641293-gd-member found.
Using host Kubernetes v1.36.4; pinning vCluster guest to v1.36.4
+ vcluster create ci-41948641293-gd-member --namespace gh-id-41948641293-global-discovery-dt --connect=false --upgrade --set controlPlane.distro.k8s.enabled=true --set controlPlane.distro.k8s.version=v1.36.4
01:50:03 fatal there is already a virtual cluster in namespace gh-id-41948641293-global-discovery-dt; creating multiple virtual clusters inside the same namespace is not supported
+ exit 1
##[error]Process completed with exit code 1.

The hub vCluster ci-41948641293-gd-hub was created and the Dynamo platform installed in it earlier in the job; creating the member vCluster ci-41948641293-gd-member in the same host namespace gh-id-41948641293-global-discovery-dt then fails with vCluster's one-cluster-per-namespace restriction. The later diagnostics (127.0.0.1:8443 connection refused, missing Grove webhook Service) are follow-on noise from this failure.

For agents

{"pr": 18161, "run_id": 41948641293, "run_attempt": 1, "head_sha": "9cdec8f76b434b695b641c1ea6368789e3a68ed8",
 "failures": [{"job": "Global Discovery Deploy Test", "job_id": 126994131613, "failed_step": "Create the member vCluster", "signature": "vcluster create: there is already a virtual cluster in namespace gh-id-41948641293-global-discovery-dt; creating multiple virtual clusters inside the same namespace is not supported", "tests": [], "log_cmd": "gh run view --job 126994131613 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 41948641293. Updated on every full-CI run of this PR.}