Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ sboms-extract/
.venv
.venv-docs

# Local MCS tools and kind state include private kubeconfigs.
deploy/global-discovery-mcs/bin/

# GPU Memory Service build artifacts
lib/gpu_memory_service/build/
lib/gpu_memory_service/*.egg-info/
Expand Down
79 changes: 79 additions & 0 deletions .github/workflows/mcs-provider-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: MCS Provider Discovery

on:
workflow_dispatch:
pull_request:
paths:
- .github/workflows/mcs-provider-e2e.yml
- .dockerignore
- .cargo/**
- Cargo.toml
- Cargo.lock
- container/Dockerfile.global-discovery-service
- deploy/global-discovery-mcs/**
- deploy/helm/charts/global-discovery-mcs/**
- lib/global-discovery-service/**
- lib/global-router/**
# GDS workspace dependencies, including the runtime's truthy dependency.
- lib/global-common/**
- lib/http-server/**
- lib/runtime/**
- lib/truthy/**
- '!**/*.md'
- '!deploy/global-discovery-mcs/test/e2e/assets/**'
- '!deploy/global-discovery-mcs/test/e2e/results/**'

concurrency:
group: mcs-provider-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

permissions:
contents: read

jobs:
discovery:
runs-on: ubuntu-24.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
provider: [dynamo, cilium]
env:
E2E_PROVIDER: ${{ matrix.provider }}
E2E_PREFIX: mcs-provider-${{ matrix.provider }}
E2E_WORK_DIR: ${{ github.workspace }}/deploy/global-discovery-mcs/bin/provider-ci
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
with:
go-version-file: deploy/global-discovery-mcs/go.mod
cache-dependency-path: deploy/global-discovery-mcs/go.sum
- name: Install Helm
env:
HELM_VERSION: v3.21.0
run: |
mkdir -p "$RUNNER_TEMP/mcs-helm"
cd "$RUNNER_TEMP/mcs-helm"
archive="helm-${HELM_VERSION}-linux-amd64.tar.gz"
curl --fail --location --silent --show-error "https://get.helm.sh/${archive}" -o "$archive"
curl --fail --location --silent --show-error "https://get.helm.sh/${archive}.sha256sum" -o checksum
sha256sum --check checksum
tar -xzf "$archive"
echo "$PWD/linux-amd64" >> "$GITHUB_PATH"
- name: Run shared Global Discovery Service assertions
run: |
set -o pipefail
make -C deploy/global-discovery-mcs e2e-discovery 2>&1 | tee mcs-provider.log
- name: Upload results and diagnostics
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: mcs-provider-${{ matrix.provider }}
path: |
mcs-provider.log
deploy/global-discovery-mcs/bin/provider-ci/reports/
deploy/global-discovery-mcs/bin/provider-ci/*.log
if-no-files-found: warn
13 changes: 11 additions & 2 deletions deploy/global-discovery-mcs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -207,13 +207,22 @@ docker-push: ## Push the image.
# (1 keeps the clusters when the run fails, always also after a pass),
# E2E_SKIP_BUILD (1 reuses the images of an earlier run), E2E_TIMEOUT.
E2E_DIR ?= test/e2e
E2E_PHASES ?= 1 2 3
export E2E_PHASES
E2E_PROVIDER ?= dynamo
E2E_PHASES ?= $(if $(filter cilium,$(E2E_PROVIDER)),3,1 2 3)
export E2E_PROVIDER E2E_PHASES

.PHONY: e2e
e2e: kind ## Run the kind end-to-end test of the registry, the controller and the Global Discovery Service.
KIND=$(KIND) $(E2E_DIR)/run.sh

.PHONY: e2e-runner-test
e2e-runner-test: ## Check runner cleanup and exit statuses without creating clusters.
bash $(E2E_DIR)/runner_test.sh

.PHONY: e2e-discovery
e2e-discovery: e2e-runner-test kind ## Run the same strict discovery cases with E2E_PROVIDER=dynamo or cilium.
E2E_PHASES=3 E2E_STRICT=1 KIND=$(KIND) $(E2E_DIR)/run.sh

.PHONY: e2e-down
e2e-down: kind ## Delete the kind clusters of the end-to-end test.
KIND=$(KIND) $(E2E_DIR)/run.sh down
Expand Down
16 changes: 14 additions & 2 deletions deploy/global-discovery-mcs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,21 @@ summary and exits non-zero on a failure.

```bash
make e2e E2E_PHASES="1 2 3"
make e2e-discovery E2E_PROVIDER=dynamo # shared discovery suite, strict mode
make e2e-discovery E2E_PROVIDER=cilium # identical cases, external MCS provider
make e2e E2E_PHASES=2 E2E_KEEP=always E2E_SKIP_BUILD=1 # iterate against kept clusters
make e2e-down
make e2e-down E2E_PROVIDER=cilium
```

Cilium mode installs Cilium 1.20.4 with MCS and EndpointSlice synchronization
in two dedicated clusters, without the bundled registry or controller. Both
providers use the same phase-3 headless Services and discovery/lifecycle
assertions. `e2e-discovery` fails on skipped cases or known gaps, and the CI
provider matrix invokes this same target. Add discovery cases once; Cilium
keeps this boundary independent of bundled MCS features. See the
[provider boundary and test settings](test/e2e/README.md).

`kind` installs into `bin/`. The Docker host needs `fs.inotify.max_user_instances`
of at least 256 for three clusters (colima: `colima ssh -- sudo sysctl -w
fs.inotify.max_user_instances=512`). `test/e2e/README.md` lists the
Expand Down Expand Up @@ -296,8 +307,9 @@ registry; it watches the EndpointSlices the controller writes:
of slices per source cluster, the local cluster included, so every
cluster's endpoints of a Service are told apart. Each slice also carries
`topology.kubernetes.io/region` when the source cluster registered a
region. The service reads it as the pool's region once ai-dynamo/dynamo#17604
lands; until then a pool's region is its cluster id.
region. The service reads it as the pool's region, using the backend fix
from [#17604](https://github.com/ai-dynamo/dynamo/pull/17604). Without the
label, a pool's region remains its cluster id.
- A pool is a Service named `<DGD name><--pool-service-suffix>` (default
`-pool-relay`) in a source cluster; the Global Router's replicas are the
ready endpoints of the Service given by `--router-namespace` and
Expand Down
1 change: 1 addition & 0 deletions deploy/global-discovery-mcs/test/conformance/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MODULE_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"

# Reuse the e2e deployment and routing helpers with a separate pair of clusters.
E2E_PROVIDER=dynamo
E2E_PREFIX="${CONFORMANCE_PREFIX:-mcs-conformance}"
E2E_WORK_DIR="${CONFORMANCE_WORK_DIR:-${MODULE_ROOT}/bin/conformance}"
E2E_CLUSTER_KEYS="hub a"
Expand Down
7 changes: 7 additions & 0 deletions deploy/global-discovery-mcs/test/e2e/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ Global Discovery Service reads the imports.
- A new assertion is a check function that returns 0 when satisfied and sets
`LAST_OBSERVED`, run through `expect` (polls up to `E2E_TIMEOUT`) or `once`
(steps that write). No fixed sleeps.
- Phase 3 is the shared provider compatibility suite. Keep its cases,
manifests and GDS image identical for Dynamo MCS and Cilium; new cases run
against both. Do not branch on the provider or use private registry APIs,
auto-export labels or provider-specific object patches to make discovery
pass. Provider installation, readiness and diagnostics belong in the setup
adapter. Run `make e2e-discovery E2E_PROVIDER=dynamo` and the same command
with `E2E_PROVIDER=cilium`; neither mode may skip discovery cases.
- Never touch anything but the clusters `<E2E_PREFIX>-hub|a|b`. Every kubectl and
helm call goes through `kc KEY` or `hm KEY`, which pass the kubeconfig file in
`bin/e2e` and the kind context, never the default kubeconfig. Never change
Expand Down
Loading
Loading