feat: update DynamoGraphDeployments for vllm_v1

[hhzhang16]

Overview:

• Added Kubernetes deployment manifests for vLLM V1 examples
• Added Kubernetes deployment documentation to the vLLM examples README to complement the existing shell script deployment instructions

Details:

• four DynamoGraphDeployment YAML files covering different serving architectures:
  • agg.yaml - Aggregated serving (single decode worker)
  • agg_router.yaml - Aggregated serving with KV routing (2 decode workers)
  • disagg.yaml - Disaggregated serving (separate prefill/decode workers)
  • disagg_router.yaml - Disaggregated serving with KV routing (2 decode + 1 prefill worker)
• Each manifest includes:
  • Frontend service with HTTP ingress and health checks
  • Vllm worker services
  • Liveness/readiness probes
  • GPU resource requests and limits

Demo:

2025-07-14-deploy-vllm-v1-disagg.mp4

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx
• precursor to Github branch hzhou/profile_vllmv1_k8s

Summary by CodeRabbit

• Documentation

  • Expanded the README with detailed instructions for Kubernetes deployment, including usage of new YAML manifests and port forwarding guidance.

• New Features

  • Added new Kubernetes deployment configurations for aggregated and disaggregated serving patterns, with and without routing.

• Refactor

  • Updated deployment manifests with new container images, working directories, and simplified startup commands.
  • Enhanced health monitoring by adding liveness and readiness probes to services.
  • Reduced resource allocations for certain services and removed deprecated services for a streamlined deployment.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

The updates introduce and revise Kubernetes deployment manifests for various vLLM serving configurations, add detailed health probes, update container images and startup commands, and remove obsolete services. Documentation is expanded with a new section on Kubernetes deployment, outlining prerequisites, usage, and deployment instructions for the provided YAML manifests.

Changes

File(s)	Change Summary
examples/vllm/README.md	Added "Kubernetes Deployment" section with instructions, prerequisites, and usage examples for new YAML manifests.
examples/vllm/deploy/agg.yaml examples/vllm/deploy/disagg.yaml examples/vllm/deploy/disagg_planner.yaml	Updated deployment names, container images, working directories, and startup commands; added health probes; removed SimpleLoadBalancer and Planner services; added componentType: worker to worker services.
examples/vllm/deploy/agg_router.yaml examples/vllm/deploy/disagg_router.yaml	Added new deployment manifests defining DynamoGraphDeployment resources for aggregated/disaggregated router setups, with detailed health probes, resource specifications, and startup commands.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Frontend (Pod)
    participant Worker (Pod)
    participant Kubernetes

    User->>Kubernetes: Apply YAML manifest (kubectl apply)
    Kubernetes->>Frontend (Pod): Schedule and start container
    Frontend (Pod)->>Frontend (Pod): Liveness/readiness probes (HTTP GET /health)
    Frontend (Pod)->>Worker (Pod): Send request for model inference
    Worker (Pod)->>Worker (Pod): Liveness/readiness probes (log check/exec)
    Worker (Pod)-->>Frontend (Pod): Return inference result
    Frontend (Pod)-->>User: Return response

Loading

Poem

🐰
Kubernetes pods now hop in style,
With health checks set to make them smile.
Old services gone, new YAMLs appear,
Images updated, the path is clear.
vLLM deployments, robust and neat—
This rabbit’s work is now complete!
🥕

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 2

🔭 Outside diff range comments (1)

examples/vllm/deploy/agg.yaml (1)

98-100: Command passed as a single argument – shell features (pipe, redirection) will not run

Because the entire string is one list element, Kubernetes will exec the container’s entrypoint with one argument that literally contains pipes and redirects.
Use /bin/sh -c so the shell interprets | tee:

-          args:
-            - "python3 components/main.py --model Qwen/Qwen3-0.6B --enforce-eager 2>&1 | tee /tmp/vllm.log"
+          args:
+            - /bin/sh
+            - -c
+            - |
+              python3 components/main.py \
+                --model Qwen/Qwen3-0.6B \
+                --enforce-eager 2>&1 | tee /tmp/vllm.log

♻️ Duplicate comments (3)

examples/vllm/deploy/disagg_router.yaml (1)

95-100: Same shell-vs-args problem as in agg_router.yaml

Apply the /bin/sh -c pattern here too; otherwise decode workers won’t start.

examples/vllm/deploy/disagg_planner.yaml (1)

52-60: Shell piping without shell – will fail

Replicate the command/args fix shown for agg_router.yaml.

examples/vllm/deploy/disagg.yaml (1)

52-60: Worker container will not start due to missing shell

Same fix as earlier manifests required.

🧹 Nitpick comments (8)

examples/vllm/README.md (2)

137-140: Prefer port-forwarding the Service, not an individual Deployment-pod

kubectl port-forward deployment/vllm-v1-disagg-frontend-<pod-uuid-info> … requires knowing an ephemeral pod name and breaks after every rollout.
Forward the stable Service instead:

-kubectl port-forward deployment/vllm-v1-disagg-frontend-<pod-uuid-info> 8080:8000
+kubectl port-forward svc/vllm-v1-disagg-frontend 8080:8000

---

146-149: Repository path looks wrong

cd ~/dynamo/examples/vllm/deploy does not match the tree (examples/vllm/deploy).
Drop the home-prefixed path to avoid copy-paste errors.

-cd ~/dynamo/examples/vllm/deploy
+cd examples/vllm/deploy

examples/vllm/deploy/disagg_router.yaml (1)

100-138: Readiness probe may never succeed if the log format changes

Coupling readiness to an exact log regex (VllmWorker.*has been initialized) is brittle. Consider exposing an HTTP /health endpoint from the worker and probing that instead.

examples/vllm/deploy/disagg_planner.yaml (1)

25-33: Probes start after 60 s with 60 s periods – slow feedback

A crashed pod may stay un-restarted for a full minute. Tighten to e.g., initialDelaySeconds: 15, periodSeconds: 10 unless cold-start time dictates otherwise.

examples/vllm/deploy/disagg.yaml (1)

25-42: Security context not set

Per team guidance (see learning on PodSecurityContext), set a pod-level runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 and override only when root is truly required.

examples/vllm/deploy/agg.yaml (3)

55-60: Minor: out=dyn looks suspicious – is this intentional?

The old command exposed HTTP output; the new one sets out=dyn, which is an internal transport that is not externally accessible. If the intention is to expose an HTTP endpoint, this should probably remain out=http.

-            - out=dyn
+            - out=http

Double-check this before shipping the manifest.

---

72-81: Race condition in worker readinessProbe

The probe greps for a one-shot log line; once it scrolls out of /tmp/vllm.log, the probe flips to Unready permanently on log rotation.
Instead, expose an HTTP /healthz in the worker or write a sentinel file after the worker is initialised and test for its existence.

---

62-62: Secret requirement not documented

envFromSecret: hf-token-secret hard-wires a secret name that must exist in the target namespace.
Add a short note in the README or the manifest’s annotations so users know to create it before applying.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6a1350c and 38eb628.

📒 Files selected for processing (6)

• examples/vllm/README.md (1 hunks)
• examples/vllm/deploy/agg.yaml (3 hunks)
• examples/vllm/deploy/agg_router.yaml (1 hunks)
• examples/vllm/deploy/disagg.yaml (4 hunks)
• examples/vllm/deploy/disagg_planner.yaml (1 hunks)
• examples/vllm/deploy/disagg_router.yaml (1 hunks)

🧰 Additional context used

🧠 Learnings (2)

examples/vllm/deploy/disagg.yaml (1)

Learnt from: julienmancuso
PR: ai-dynamo/dynamo#1474
File: deploy/cloud/operator/internal/controller/dynamocomponent_controller.go:1302-1306
Timestamp: 2025-06-11T21:18:00.425Z
Learning: In the Dynamo operator, the project’s preferred security posture is to set a Pod-level `PodSecurityContext` with `runAsUser`, `runAsGroup`, and `fsGroup` all set to `1000`, and then selectively override the user at the individual container level (e.g., `RunAsUser: 0` for Kaniko) when root is required.

examples/vllm/deploy/agg.yaml (1)

Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/configs/agg_tp_1_dp_8.yaml:31-38
Timestamp: 2025-07-01T15:33:53.262Z
Learning: In fault tolerance test configurations, the `resources` section under `ServiceArgs` specifies resources per individual worker, not total resources for all workers. So `workers: 8` with `gpu: '1'` means 8 workers × 1 GPU each = 8 GPUs total.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Build and Test - vllm

🔇 Additional comments (4)

examples/vllm/deploy/agg_router.yaml (2)

86-93: Non-standard GPU resource key

Use the standard nvidia.com/gpu resource unless DynamoGraphDeployment mutates it internally. Sticking to the canonical key avoids scheduler mis-accounting on vanilla clusters.

---

52-60: args string uses shell features but no shell is invoked

| tee /tmp/vllm.log requires a shell. Because only args is supplied, Kubernetes passes the whole string to the image ENTRYPOINT.
Unless the image’s ENTRYPOINT is /bin/sh -c, the container will fail.

-          args:
-            - "python3 components/main.py --model Qwen/Qwen3-0.6B --enforce-eager 2>&1 | tee /tmp/vllm.log"
+          command: ["/bin/sh", "-c"]
+          args:
+            - python3 components/main.py --model Qwen/Qwen3-0.6B --enforce-eager 2>&1 | tee /tmp/vllm.log

Likely an incorrect or invalid review comment.

examples/vllm/deploy/agg.yaml (2)

52-60: Verify that the working directory exists inside the new image

The image was bumped and the working directory changed from /workspace/examples/vllm_v1 to /workspace/examples/vllm.
If that directory is missing, dynamo will fail to locate model/asset files at container start-up.

Please confirm the new image layout or revert the path.

---

30-38: Replace exec readinessProbe with httpGet to avoid relying on jq

• File: examples/vllm/deploy/agg.yaml (lines 30–38)
• Verify whether the nvcr.io/nvidian/nim-llm-dev/vllm_v1-runtime:dep-216.4 image includes jq; if not, the current probe will always fail.

-      readinessProbe:
-        exec:
-          command:
-            - /bin/sh
-            - -c
-            - 'curl -s http://localhost:8000/health | jq -e ".status == \"healthy\""'
+      readinessProbe:
+        httpGet:
+          path: /health
+          port: 8000
       initialDelaySeconds: 60
       periodSeconds: 60
       timeoutSeconds: 30

Using an httpGet probe removes the external-binary dependency. If you truly need JSON parsing, consider baking jq into a custom image, but that increases attack surface.

[biswapanda]

Approved to unblock.
PTAL the liveness comment

[nnshah1]

health and ready checks will be revised when health and ready endpoints are exposed from rust runtime

[waynepan]

fyi the changes in the .yaml seem to point to private images

          image: nvcr.io/nvidian/nim-llm-dev/vllm_v1-runtime:dep-216.4