fix(inference): stop a managed vLLM install the host GPU cannot serve

[laitingsheng]

Summary

A quantized checkpoint whose GPU requirement the host cannot meet used to download in full and then crash-loop the nemoclaw-vllm container until the 30-minute load timeout expired, with no actionable message from onboarding. The managed install path now compares each model's minimum GPU compute capability against the host before the image pull and the model download, and the readiness poller stops a container that keeps restarting instead of waiting out the timeout.

Related Issue

Fixes #8307

Changes

• Record minComputeCapability on VllmModelDef, on the two-digit scale vLLM reports. Both FP8 checkpoints require 8.9, deepseek-v4-flash and the Ultra NVFP4 recipe require 10.0, and the Spark NVFP4 recipe requires 12.1. The bf16 DeepSeek-R1 distill has no minimum.
• Add readGpuComputeCapabilities, formatComputeCapability, and computeCapabilityPreflight to the managed vLLM module. The generic Linux and DGX Spark profiles launch with --gpus all, so the check judges the host by its weakest reported GPU. A host that reports no compute capability is not blocked.
• Run the check in runVllmInstall immediately after the Docker prerequisites, ahead of the container-ownership check, the image pull, and hf download.
• Watch the container restart count while waiting for the API. The single-host container runs under --restart unless-stopped, so docker ps keeps reporting a crash-looping container as running and the existing exited-container check never fires. The install now stops after three restarts and prints the container log tail.
• Document the new check and add a minimum-capability column to the managed model table.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: docs/inference/set-up-vllm.mdx, docs/inference/set-up-vllm-on-two-dgx-stations.mdx. The review found that the page claimed onboarding stops on a capability mismatch, which is true only for non-interactive runs; that the restart threshold was unstated; that the crash-loop text was attributed to the managed container in general rather than the single-host container; and that the distributed Station page shares the same entry point without saying so. Commit ba06343a9 applies those corrections. Revision 34a616316 adds the final clarification that reaching the restart threshold stops both the container and the install.
• Agent: Claude Code

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run --project cli src/lib/inference/ — 77 files, 1589 tests passed; npm run typecheck:cli — clean; npm run checks:repository — passed
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

• New Features

  • Added compatibility checks to ensure selected models can run on available GPUs before downloads begin.
  • Added minimum GPU capability requirements for supported managed models.
  • Added detection of repeated container restarts during vLLM startup, with clearer failure reporting.

• Documentation

  • Documented GPU requirements, validation behavior, failure handling, and readiness monitoring for managed vLLM deployments, including distributed installations.

• Tests

  • Added coverage for GPU validation, model compatibility, capability detection, and startup crash-loop handling.

[coderabbitai]

📝 Walkthrough

Walkthrough

Managed vLLM now records model GPU capability requirements, validates detected GPUs before downloads, and stops readiness polling after repeated container restarts. Tests cover capability handling, installation behavior, and restart detection. Documentation describes the new checks and managed-model requirements.

Changes

Managed vLLM validation

Layer / File(s)	Summary
Model capability requirements src/lib/inference/vllm-models.ts, docs/inference/set-up-vllm.mdx	VllmModelDef now supports optional minimum compute capabilities. Requirements were added to five managed models and documented.
Capability detection and installation preflight src/lib/inference/vllm.ts, src/lib/inference/vllm-compute-capability.test.ts, docs/inference/set-up-vllm.mdx, docs/inference/set-up-vllm-on-two-dgx-stations.mdx	GPU capabilities are parsed and formatted. Installation rejects incompatible models before image or model operations. Tests cover unsupported, supported, undetected, and weakest-GPU cases.
Startup restart watchdog src/lib/inference/vllm.ts, src/lib/inference/vllm-compute-capability.test.ts, docs/inference/set-up-vllm.mdx	Readiness polling inspects container restart counts and fails after three startup restarts. Tests cover repeated restarts and stable containers.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant InstallFlow
  participant GPUDetection
  participant ModelRegistry
  participant Docker
  InstallFlow->>GPUDetection: Read GPU compute capabilities
  GPUDetection-->>InstallFlow: Return detected capabilities
  InstallFlow->>ModelRegistry: Validate model requirement
  ModelRegistry-->>InstallFlow: Return compatibility result
  InstallFlow->>Docker: Start vLLM container when compatible
  Docker-->>InstallFlow: Return restart count during readiness polling
  InstallFlow->>Docker: Stop container after three startup restarts

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements GPU capability preflight, prevents downloads on incompatible hosts, and stops repeated vLLM restarts as required by [#8307].
Out of Scope Changes check	✅ Passed	The implementation, tests, model requirements, and documentation directly support the linked issue objectives without unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: stopping managed vLLM installation when host GPUs cannot serve the selected model.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/vllm-compute-capability-preflight

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8314.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 34a6163 in the fix/vllm-compute-cap... branch remains at 96%, unchanged from commit 3312899 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 34a6163 in the fix/vllm-compute-cap... branch remains at 81%, unchanged from commit 3312899 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 3312899	fix/vllm-compute-cap... 34a6163	+/-
src/lib/credentials/store.ts	56%	55%	-1%
src/lib/inferen.../vllm-models.ts	81%	80%	-1%
src/lib/inference/vllm.ts	89%	89%	0%
src/lib/onboard...-transaction.ts	94%	94%	0%
src/lib/sandbox...rce-identity.ts	87%	87%	0%
src/lib/tunnel/services.ts	80%	80%	0%
src/lib/shields/index.ts	73%	74%	+1%

---

_{Updated August 05, 2026 11:18 UTC}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/inference/vllm-compute-capability.test.ts`:
- Around line 303-332: Update the two tests around installVllm to cover the
restart-limit boundary: change the failure case in “stops a restarting container
instead of waiting out the load timeout (`#8307`)” to simulate three restarts and
expect the message mentioning three restarts, and rename the success case to
describe waiting while restarts remain below the limit while keeping its
two-restart setup and successful result.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 074407c3-0d50-4ada-bfcd-bc40d363b03a

📥 Commits

Reviewing files that changed from the base of the PR and between 3a39ff3 and d0aae96.

📒 Files selected for processing (4)

• docs/inference/set-up-vllm.mdx
• src/lib/inference/vllm-compute-capability.test.ts
• src/lib/inference/vllm-models.ts
• src/lib/inference/vllm.ts

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections differ; severity counts match.

5 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

• GPU compute capability at docs/inference/set-up-vllm.mdx:135: primary classified it as define; the second opinion classified it as established.
• compute capability preflight at src/lib/inference/vllm-compute-capability.test.ts:118: selected only by the second-opinion lane as justified.
• crash-loop watchdog at src/lib/inference/vllm-compute-capability.test.ts:252: selected only by the second-opinion lane as justified.
• minComputeCapability at src/lib/inference/vllm-models.ts:92: selected only by the second-opinion lane as justified.
• restart count at docs/inference/set-up-vllm.mdx:209: selected only by the second-opinion lane as established.

3 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

• gpu-e2e: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• managed-image-multiarch-startup: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• vllm-docker-storage: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• define — GPU compute capability at docs/inference/set-up-vllm.mdx:135: Define the term at first use through the surrounding explanation of reported and required values; retain this term consistently for the preflight.
• justified — startup restart limit at src/lib/inference/vllm-compute-capability.test.ts:303: Retain the modifier because it distinguishes the restart-count threshold from the load timeout.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: inference-routing, network-policy

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Security review for revision 34a616316: PASS — no findings.

Category	Result	Evidence
Secrets and credentials	PASS	No credential source, storage, forwarding, or logging changes.
Input validation and data sanitization	PASS	nvidia-smi output is accepted only as decimal compute-capability lines and converted to bounded numeric comparisons.
Authentication and authorization	PASS	The change does not alter provider authentication or authorization decisions.
Dependencies and third-party libraries	PASS	No dependency is added or updated.
Error handling and logging	PASS	Capability mismatch stops before downloads with non-sensitive diagnostics; malformed or unavailable capability output preserves the existing vLLM fallback path.
Cryptography and data protection	PASS	No cryptographic or stored-data behavior changes.
Configuration and security headers	PASS	Model minima are declarative registry values, and no sandbox or network-policy configuration is broadened.
Security testing	PASS	Tests cover below-minimum rejection, boundary acceptance, missing or invalid capability output, mixed-GPU hosts, registry coverage, and the restart threshold.
System security	PASS	Repeated startup crashes now stop the managed container after three restarts rather than allowing a prolonged crash loop; no privilege boundary changes.

All commits are GitHub Verified and the PR includes contributor DCO. Focused vLLM tests passed 8/8, the previously failing package-contract test passed 8/8 after incorporating the fixture fix from current main, CLI type-check passed, repository hooks passed, and the documentation build completed with 0 errors. Fresh CI is running.

[apurvvkumaria]

Final gate refresh for the current branch revision: all 40 reported checks pass, including protected E2E, both advisor lanes, CodeQL, DCO, documentation receipt, package contracts, and the aggregate CI gate. There are no unresolved review threads or actionable automated findings. The PR remains unmerged because an independent approval is still required.