Skip to content

feat(dynamo): add Kubernetes DGD support - #10

Draft
jthomson04 wants to merge 23 commits into
jthomson04/dynamo-slurm-wandbfrom
agent/dynamo-k8s-on-slurm
Draft

feat(dynamo): add Kubernetes DGD support#10
jthomson04 wants to merge 23 commits into
jthomson04/dynamo-slurm-wandbfrom
agent/dynamo-k8s-on-slurm

Conversation

@jthomson04

@jthomson04 jthomson04 commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • stack the Kubernetes-specific Dynamo work from feat: Dynamo K8s Integration NVIDIA-NeMo/RL#2990 on top of the managed Dynamo/Slurm implementation in feat(dynamo): add managed Slurm generation with W&B telemetry NVIDIA-NeMo/RL#3391
  • add nrl-k8s lifecycle support for suspended RayJobs, DynamoGraphDeployments, readiness, ownership, reparenting, and cleanup
  • add a strict external-DGD runtime that discovers and freezes the vLLM worker fleet while leaving serving lifecycle ownership to Kubernetes
  • allocate all Ray GPUs to training for external Dynamo and support collective weight synchronization without an inference Ray cluster
  • carry forward the V1, V2, V3, and V5 GB300 Kubernetes recipes and document the ownership/refit model

Validation

  • 275 nrl-k8s unit tests passed
  • 142 focused Dynamo and weight-sync unit tests passed on the latest PR 3391 head
  • Pyrefly: 0 errors across modified core modules
  • Ruff lint and format checks passed
  • all four Dynamo recipe/infra pairs pass nrl-k8s check
  • all four Dynamo recipe/infra pairs pass RayJob dry-run rendering
  • live V1 brought up the DGD and Ray resources and reached NeMo-RL model initialization; that smoke test exposed and drove fixes for strict recipe fields and external-Dynamo weight-sync construction

The live V1 two-step completion and V3 smoke remain to be captured. The validation runner lost namespace RBAC overnight, so this remains a draft until those runs can be observed end to end.

Stack

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 13, 2026
jthomson04 and others added 17 commits August 15, 2026 20:23
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Co-authored-by: Terry Kong <terrycurtiskong@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/dynamo-slurm-wandb branch from bebf9c7 to 39dc01d Compare August 16, 2026 03:25
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the agent/dynamo-k8s-on-slurm branch from bdbc8d4 to 136ff0f Compare August 17, 2026 16:55
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/dynamo-slurm-wandb branch 2 times, most recently from 49e7f97 to e4c8b79 Compare August 18, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant