Repository navigation
feat(router): add a Global Discovery HTTP service POC - #16714
Closed
atchernych wants to merge 1 commit into
Closed
atchernych wants to merge 1 commit into
atchernych wants to merge 1 commit into
Conversation
Serve the KV DC Relay pools found in Karmada-imported EndpointSlices as JSON on GET /pools, so the Global Router can learn its pools from multi-cluster services instead of a static file. The service watches EndpointSlices labelled by Karmada's EndpointSlice controller and keeps one index entry per slice, replaced as a whole on relist. A Relay exported as `relay-<dgd>` maps to pool (<cluster>, <namespace>, <dgd>); the cluster comes from the imported slice name because Karmada adds no source-cluster label. The binary sits behind the new `global-discovery-poc` feature, so default builds are unchanged. A Dockerfile and a hub-cluster manifest are included. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
Contributor
❌ Dynamo PR CI failed — run 38987872826 (attempt 1) on
|
| Other | Jobs |
|---|---|
| changed-files | ❌ 1 |
Failure details
1 job failed: changed-files rejects two new files that no CI filter covers, so every downstream job was skipped.
❌ changed-files: files not covered by any CI filter
Job: changed-files · Failed step: Check for changes · Logs: gh run view --job 117263440876 -R ai-dynamo/dynamo --log-failed
##[error]The following files are not covered by any CI filter:
container/Dockerfile.global-discovery-poc
container/global-discovery-poc.yaml
Please add these paths to .github/filters.yaml
See .github/FILTERS.md for documentation
##[error]Process completed with exit code 1.
The uncovered-files check failed because container/Dockerfile.global-discovery-poc and container/global-discovery-poc.yaml don't match any filter in .github/filters.yaml. Adding these paths to a filter (see .github/FILTERS.md) should fix it.
For agents
{"pr": 16714, "run_id": 38987872826, "run_attempt": 1, "head_sha": "77fb662dca4fb29d860d411c0973be57f99e155b",
"failures": [{"job": "changed-files", "job_id": 117263440876, "failed_step": "Check for changes", "signature": "files not covered by any CI filter: container/Dockerfile.global-discovery-poc, container/global-discovery-poc.yaml", "tests": [], "log_cmd": "gh run view --job 117263440876 -R ai-dynamo/dynamo --log-failed"}]}Posted automatically by Devin for run 38987872826. Updated on every full-CI run of this PR.
atchernych
added a commit
that referenced
this pull request
Oct 20, 2026
Fill in the router side of Kubernetes mode: - mcs.rs: label selectors and naming rules for standard MCS and Karmada. Karmada's source cluster comes from the slice name `imported-<cluster>-<source slice>`. The `work.karmada.io/namespace` annotation names the source cluster only on the Karmada control plane; copies that Karmada propagates into members carry the destination. - McsPoolDirectory: one watch feeds a shared cache. Each slice event updates only its pool, a relist rebuilds everything once, `get` reads the cache, and `list_and_watch` sends `Added` per pool, a `Resync`, then changes. A bound Relay address stays while it is Ready. - TrustedNetworkAuthenticator, MeshAuthenticator, address_identity and McsPoolDiscovery::build. - The body of pool_discovery::admit, with tests. McsImplementation is no longer non_exhaustive, so dynamo-llm handles every implementation. The Relay side (EndpointSliceReplicaDirectory) stays a sketch. The naming rules and the Karmada findings come from a test of the POC service in #16714 on two EKS clusters joined to Karmada v1.19. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
atchernych
added a commit
that referenced
this pull request
Oct 23, 2026
Adds the Global Discovery Service, an HTTP service that PoolRelays and Global Routers call instead of reading Kubernetes themselves: - POST /pools registers a pool. A PoolRelay calls it on a timer, so each call is also the pool's heartbeat. - GET /routers lists the Global Router replicas a PoolRelay connects to. - GET /pools lists pools for routers. GET /healthz is OK while the service can answer from its store, which for MCS means its cache. The JSON matches the dynamo-global-discovery-client crate. The service works through the pool_discovery traits, with two stores: - memory (the default): registrations in this process, each expiring lease_duration after its last registration; router replicas come from --router. - karmada and mcs: pools and router replicas from the EndpointSlices that MCS imports. Pools come from McsPoolDirectory, and router replicas from EndpointSliceReplicaDirectory, which this change implements. These stores take no registrations (501), because an MCS pool is announced by its ServiceExport. Both EndpointSlice directories now share one cache (kubernetes/cache.rs) and one watch-stream helper (stream.rs). PoolRelay Services are named <DGD name><suffix>, with -pool-relay as the default suffix. This replaces the relay-<DGD name> prefix used in #16714. The binary needs the global-discovery feature: cargo run -p dynamo-llm --features global-discovery --bin global-discovery Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
Contributor
Author
atchernych
added a commit
that referenced
this pull request
Oct 27, 2026
…l Discovery Service Copied from the POC in #16714 and updated for this crate's binary: - container/Dockerfile.global-discovery-service builds global-discovery-service from dynamo-global-discovery-service into a CPU-only Debian image, as the POC did for its own binary. - container/global-discovery-service.yaml deploys it on the hub with the Karmada store, the router Service global-router and its port http, as the test deployment in #17278 does. It needs only read access to EndpointSlices. Logging is set with DYN_LOG, which the service reads, instead of RUST_LOG. The Dockerfile is owned by the global router and ops teams, like other component Dockerfiles, and the manifest by the global router team. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anna Tchernych <atchernych@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is the Karmada Specific implementation. This will serve as a feed into the more generic Kubernetes implementation in #15949.
Fixes DEP-1290
POC of a Global Discovery HTTP service for the multi-cluster Global Router. Instead of a static pool list, the router can learn its pools from Kubernetes multi-cluster services (MCS) imported by Karmada.
GET /pools.GET /healthzturns ready after the first full list.relay-<dgd>maps to pool(<cluster>, <namespace>, <dgd>). Karmada adds no source-cluster label, so the cluster comes from the imported slice name (imported-<cluster>-<service>-<suffix>).global-discovery-pocfeature ofdynamo-llm, so default builds are unchanged.container/Dockerfile.global-discovery-pocandcontainer/global-discovery-poc.yaml(read-only EndpointSlice access, Deployment, Service).Stacked on #15579. It is a working stand-in for the Kubernetes
PoolDirectorysketched in #15949; the plan is to move this logic behind that trait later.Not in this PR: the router reading
/pools.Validation
cargo test -p dynamo-llm --no-default-features --features global-discovery-poc --bin global-discovery-poc: 9 tests pass (site parsing, non-Relay services skipped, not-ready pools, IPv6 URLs, delete, relist).cargo clippyon the binary: no warnings in the new file.cargo fmt --check: clean.derived-relay-<dgd>with animported-<cluster>-...slice, which is the input this service reads. The service itself is not yet deployed there.🤖 Generated with Claude Code