Skip to content

fix(models): order dual-Ada startup - #328

Open
nvddr wants to merge 2 commits into
mainfrom
agent/dual-ada-startup-order
Open

fix(models): order dual-Ada startup#328
nvddr wants to merge 2 commits into
mainfrom
agent/dual-ada-startup-order

Conversation

@nvddr

@nvddr nvddr commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • make shared model-server ordering profile-aware
  • start VLM, 8B LLM, STT, then the dedicated 30B LLM on dual Ada
  • retain large-model-first startup for single-GPU profiles

Why

The dual-Ada topology benefits from settling shared GPU 0 services before using the dedicated GPU 1 model as the final readiness gate. Single-GPU profiles still need the 30B model to compile against an empty device.

Validation

  • uv run --project tests pytest -q tests/test_model_servers.py
  • ruff check on changed Python files
  • SPDX header check

Signed-off-by: Devdeep Ray <devdeepr@Nvidia.com>
@nvddr
nvddr force-pushed the agent/dual-ada-startup-order branch from 48eb125 to b97cf4e Compare July 31, 2026 20:54
@nvddr

nvddr commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

Review outcome: ready for review

The branch is rebased on current main, and the earlier topology/test blockers are resolved:

  • The dual-Ada default stack now starts VLM and embeddings on GPU 0 before STT and the 30B agent LLM on GPU 1.
  • Single-GPU profiles retain their existing 30B-before-smaller-vLLM order.
  • The independently selected Omni stack, incompatible-stack cleanup, credential enforcement, embedding configuration, and stack-selection CLI behavior are preserved.
  • Tests cover dual-Ada/default, both single-GPU profiles/default, and Omni behavior; documentation now names the current services.

Validation:

  • Focused isolated suite: 20 passed (tests/test_model_servers.py)
  • Ruff: passed
  • Fresh GitHub CI: all checks passed (Python 3.11/3.12, docs build/publish, Ruff, lock, SPDX, DCO, and CodeQL)

No unresolved review threads remain. The PR is ready for maintainer review and merge.

@nvddr
nvddr deployed to github-pages August 10, 2026 22:00 — with GitHub Actions Active
github-actions Bot added a commit that referenced this pull request Aug 10, 2026
@nvddr
nvddr marked this pull request as ready for review August 10, 2026 22:11
@yanziz-nvidia
yanziz-nvidia self-requested a review August 12, 2026 03:47

@yanziz-nvidia yanziz-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by yanziz-reviewer-bot

Summary

Orders dual-Ada startup so VLM and embeddings (GPU 0) settle before STT and the 30B LLM (GPU 1), leaving the largest model as the final readiness gate. Single-GPU profiles (spark, 96G_blackwell) keep the existing 30B-first order. CI green (ruff, pytest 3.11 + 3.12, CodeQL, DCO, docs); DEPENDENCIES.md updated.

Legend: 🚫 Blocker · 💡 Suggestion · 🔍 Nit

Finding
🚫 None
💡 None
🔍 agent-samples/model-servers/main.py:11-16The "Servers started" table still lists stt → agent-llm → vlm → embedding, which now reflects only the single-GPU profiles. A one-line note on the dual-Ada order keeps the docstring authoritative.

Actionables (for bots — copy-paste-ready for AI)

Fix if it makes sense in context — these are agent-generated suggestions, not human-vetted obligations. Skip anything that's wrong, already addressed, or not worth the churn.

  • agent-samples/model-servers/main.py:11 — Annotate the "default / --vlm-llm-stack" table to note startup order varies by profile (dual-Ada: vlm → embedding → stt → agent-llm; single-GPU: stt → agent-llm → vlm → embedding), so the header stays accurate after this change.

@yanziz-nvidia yanziz-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve merge conflicts fist; LGTM otherwise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants