ci: add Kubernetes deploy test to nightly by knarangN · Pull Request #12090 · ai-dynamo/dynamo

knarangN · 2026-07-24T00:00:06Z

Overview:

Add a Kubernetes deployment-test lane to nightly CI for vLLM, SGLang, and TensorRT-LLM runtime images.

Details:

build and publish the operator with a nightly-specific immutable tag
copy each nightly runtime image to ACR using the corresponding build output
create a nightly-prefixed vCluster and optionally provision the shared model-cache PVC
run the supported agg baseline for vLLM, SGLang, and TensorRT-LLM
always tear down the nightly vCluster after all three deployment lanes finish
make builder cleanup account for the operator build and make Slack notification wait for deployment cleanup

The common agg case is intentional: it validates scheduled *-nightly artifacts against the same known deployment contract used post-merge, isolating image and dependency drift from profile-specific behavior. Nightly-only profiles are deferred to a follow-up after these three baseline lanes have trusted scheduled-runtime evidence.

Where should the reviewer start?

Start in .github/workflows/nightly-ci.yml at the NIGHTLY K8S DEPLOYMENT section. The key review points are the three copy jobs, the three reusable deploy-test calls, nightly-specific naming, and the cleanup dependency graph.

Related Issues

🚫 This PR is NOT linked to a GitHub issue:

Confirmed — no related GitHub issue

Internal tracking: OPS-7826

Blocks the nightly infrastructure portion of OPS-7603 / PR #12085.

Validation

Current merged head: 06d59f7cc4

merged current origin/main at d698bebc73
added an explicit valid host namespace, nightly-ci-${{ github.run_id }}, after the first manual run exposed an invalid auto-generated cleanup label for the tests/ops-7826-nightly-k8s-support branch
git diff --check passed; YAML parsing and Kubernetes label-shape validation passed
codespell, YAML, case-conflict, merge-conflict, executable, line-ending, and trailing-whitespace hooks passed
repository-wide pytest-marker-report found zero missing marker sets but local collection stopped on the known port allocator exhaustion after 100 retries

Previous head b467ddaccd was validated by trusted PR run 30056153965: all 75 jobs passed, including the SGLang agg deployment, broader deployment matrix, and cleanup.

Runtime validation boundary

The non-release manual workflow run 30403849220 on previous head 82a664895b completed with failure:

the operator and all three nightly runtime image-copy jobs succeeded
deploy-operator failed before vCluster creation because the auto-generated Kubernetes cleanup-label value ended in -
all three new framework deployment tests were therefore skipped; deployment cleanup still succeeded
the remaining runtime-test failures reproduced existing scheduled-main nightly failures and did not originate in this PR's workflow diff

Replacement non-release manual workflow run 30420741044 was dispatched on 06d59f7cc4 with tests enabled and release/GitLab automation disabled. Until it completes, the namespace fix and all three deployment lanes remain runtime-pending. The first scheduled run after merge is still required to prove the exact cron-triggered path.

Signed-off-by: Kavita Narang <knarang@nvidia.com>

copy-pr-bot · 2026-07-24T00:00:10Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

knarangN · 2026-07-24T00:01:25Z

/ok to test 4762be3

coderabbitai · 2026-07-24T00:03:14Z

Walkthrough

Changes

Nightly Kubernetes deployment

Layer / File(s)	Summary
Kubernetes deployment and validation `.github/workflows/nightly-ci.yml`	Adds gated jobs to build and push the operator, copy the nightly SGLang image, deploy into vCluster, and run the SGLang deploy-test.
Cleanup and notification ordering `.github/workflows/nightly-ci.yml`	Makes builder cleanup wait for the operator job and Slack notification wait for the SGLang deploy-test.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly matches the main change: adding a Kubernetes deploy test to nightly CI.
Description check	✅ Passed	The description follows the template with Overview, Details, reviewer start, and the required Related Issues section.

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: Kavita Narang <knarang@nvidia.com>

knarangN · 2026-07-24T00:25:38Z

/ok to test b467dda

Signed-off-by: Kavita Narang <knarang@nvidia.com>

nv-tusharma · 2026-07-28T22:12:22Z

LGTM, feel free to schedule a nightly pipeline on your branch to validate this works as expected.

…y-k8s-support

Signed-off-by: Kavita Narang <knarang@nvidia.com>

knarangN · 2026-07-29T03:52:09Z

/ok to test 06d59f7

Signed-off-by: Kavita Narang <knarang@nvidia.com>

knarangN · 2026-08-01T02:22:17Z

/ok to test 3f7892d

dyn-3691-extract-shared-target-pid-cuda-customstorage-operation-layer * 'main' of https://github.com/ai-dynamo/dynamo: (50 commits) docs(cli): correct removed vLLM prefill-worker flag reference (#12581) docs(operator): reserve webhook Ignore for emergencies (#12563) ci(docs): make previews and checks match what actually publishes (#12339) refactor(vllm): organize custom encoder modules (#12416) feat(llm): Select reasoning output field via env var (#11464) feat(runtime): add TLS support to TCP request plane (#10921) fix: convert conditional disagg sglang warning to httperror 400 (#12578) feat(operator): add runtime feature gates (#12421) refactor(runtime): extract PushRouter transport seam behind StreamingDispatch trait (#12447) feat(replay): add deterministic canonical offline reports (#12363) build: bump ModelExpress to 0.5.0(OPS-7978) (#12455) fix(mocker): use logical KV tokens for decode timing (#12583) fix(examples): update Triton example for CUDA 13 + fix libdcgm copy (DYN-3697) (#12577) refactor(operator): implement composition-first DGD reconciliation (#12283) feat(frontend): add basetenkenizer backend (#12376) fix(profiler): configure rapid mocker without planner (#12573) docs(vllm): correct worker-role flags and document --kv-transfer-config (#12568) ci: add Kubernetes deploy test to nightly (#12090) fix(container): reuse pinned protoc in runtime image (#12535) feat(self-host): flip DYN_SELF_HOST_METADATA default to ON (gh-8749) (#11417) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

The nightly deploy-test jobs added in #12090 call shared-deploy-test.yml with `image_suffix`, but that workflow declares a required `image_tag` input and no `image_suffix`. GitHub rejects the whole file at parse time: .github/workflows/nightly-ci.yml (Line: 610, Col: 11): Input image_tag is required, but not provided while calling. .github/workflows/nightly-ci.yml (Line: 614, Col: 21): Invalid input, image_suffix is not defined in the referenced workflow. That is a hard "Invalid workflow file" error, so no job in nightly-ci runs at all -- the scheduled nightly has been dead since #12090 merged. Pass the full ACR tag from the corresponding copy-to-acr job, matching how pr.yaml and post-merge-ci.yml call the same workflow. shared-deploy- test.yml interpolates the value directly as `ai-dynamo/dynamo:${{ inputs.image_tag }}`, so a bare suffix such as `vllm-nightly` would not have resolved to a real image either. Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>

ci: add Kubernetes deploy test to nightly

4762be3

Signed-off-by: Kavita Narang <knarang@nvidia.com>

knarangN requested a review from a team as a code owner July 24, 2026 00:00

pull-request-size Bot added the size/M label Jul 24, 2026

github-actions Bot added ci Issues/PRs that reference CI build/test actions labels Jul 24, 2026

knarangN mentioned this pull request Jul 24, 2026

test(deploy): add SGLang JSON logging coverage #12085

Merged

1 task

dynamo-review-agent Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread .github/workflows/nightly-ci.yml

Comment thread .github/workflows/nightly-ci.yml

This comment has been minimized.

Sign in to view

ci: provision and clean nightly deploy environment

b467dda

Signed-off-by: Kavita Narang <knarang@nvidia.com>

pull-request-size Bot added size/L and removed size/M labels Jul 24, 2026

dynamo-review-agent Bot approved these changes Jul 24, 2026

View reviewed changes

knarangN requested review from dmitry-tokarev-nv, nv-tusharma and sara4dev July 24, 2026 01:11

knarangN enabled auto-merge (squash) July 24, 2026 03:40

nv-tusharma reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/nightly-ci.yml Outdated

nv-tusharma reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/nightly-ci.yml Outdated

nv-tusharma reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/nightly-ci.yml

nv-tusharma reviewed Jul 28, 2026

View reviewed changes

Comment thread .github/workflows/nightly-ci.yml

knarangN added 2 commits July 28, 2026 13:03

ci: address nightly deploy review feedback

6cd9a31

Signed-off-by: Kavita Narang <knarang@nvidia.com>

Merge origin/main into tests/ops-7826-nightly-k8s-support

82a6648

Signed-off-by: Kavita Narang <knarang@nvidia.com>

knarangN added 2 commits July 28, 2026 20:47

Merge remote-tracking branch 'origin/main' into tests/ops-7826-nightl…

baf1d8b

…y-k8s-support

ci: use valid namespace for nightly deploy

06d59f7

Signed-off-by: Kavita Narang <knarang@nvidia.com>

Merge origin/main into tests/ops-7826-nightly-k8s-support

3f7892d

Signed-off-by: Kavita Narang <knarang@nvidia.com>

nv-tusharma approved these changes Aug 3, 2026

View reviewed changes

knarangN merged commit 7739f22 into main Aug 3, 2026
112 checks passed

knarangN deleted the tests/ops-7826-nightly-k8s-support branch August 3, 2026 18:56

nv-tusharma mentioned this pull request Aug 4, 2026

fix(ci): pass image_tag to nightly deploy tests #12637

Closed

pvijayakrish mentioned this pull request Aug 4, 2026

fix(ci): pass image_tag to shared-deploy-test in nightly (8015) #12642

Merged

Navigation Menu

Uh oh!

ci: add Kubernetes deploy test to nightly - #12090

ci: add Kubernetes deploy test to nightly#12090
knarangN merged 7 commits into
mainfrom
tests/ops-7826-nightly-k8s-support

knarangN commented Jul 24, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

knarangN commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

knarangN commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nv-tusharma commented Jul 28, 2026

Uh oh!

knarangN commented Jul 29, 2026

Uh oh!

knarangN commented Aug 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

knarangN commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview:

Details:

Where should the reviewer start?

Related Issues

Validation

Runtime validation boundary

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

knarangN commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

knarangN commented Jul 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nv-tusharma commented Jul 28, 2026

Uh oh!

knarangN commented Jul 29, 2026

Uh oh!

knarangN commented Aug 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

knarangN commented Jul 24, 2026 •

edited

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading