Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,8 @@ xr-ai-models (agent-sdk/xr-ai-models/)
seam: reasoning-field aliasing (nano_v3 →
`reasoning`, nemotron_v3 → `reasoning_content`), `chat_template_kwargs`
plumbing for `enable_thinking` / `thinking_budget`, and built-in presets
for the in-tree services. Future backends (LiteLLM, vendor SDKs)
for the in-tree services, including distinct Cosmos3 Nano Reasoner and
Cosmos-Reason1 VLM profiles. Future backends (LiteLLM, vendor SDKs)
plug in as new `kind`s in `_factory.py::make_*` without touching the
protocols or callers. Workers depend on this instead of rolling their
own httpx wrappers. Profiles may separate adapter, endpoint, and deployment
Expand Down Expand Up @@ -199,7 +200,9 @@ xr-ai-vllm (utils/xr-ai-vllm/)
`--stop` flow. Besides `serve` / `stop_persistent_servers`, exposes the
shared wrapper helpers `resolve_model_cache`, `load_config`, `setup_hf_env`,
and `gpu_compute_major` (all stdlib-only; pyyaml is imported function-locally
inside `load_config` so the `--stop` path stays dependency-free).
inside `load_config` so the `--stop` path stays dependency-free). Persistent
Docker containers and pip processes carry launch fingerprints; stale
repository-owned runtimes are replaced before reuse.

xr-ai-vad (utils/xr-ai-vad/)
└── numpy >=1.24
Expand Down Expand Up @@ -371,12 +374,15 @@ xr-ai-tests (tests/)
`video-mcp-server` rather than redeclared here.

vlm-server (services/vlm-server/)
└── vllm >=0.12.0
└── vllm >=0.23.0
└── pyyaml >=6.0
└── hf-transfer >=0.1.4
└── xr-ai-logging [editable: ../../utils/xr-ai-logging]
└── xr-ai-vllm [editable: ../../utils/xr-ai-vllm]
Model: nvidia/Cosmos-Reason1-7B (Qwen2.5-VL architecture, vLLM).
Default model: the text-output Reasoner from nvidia/Cosmos3-Nano. Standard
vLLM serving loads only the Reasoner; the Generator's vLLM-Omni path is
outside this service. nvidia/Cosmos-Reason1-7B remains configurable with
the cosmos_vlm client preset.
Wrapper Popens `vllm serve` so the launcher's killpg() does not reach
vLLM — model survives stack restarts (see docs/changelog.md 2026-05-05).
vllm_backend: pip|docker — pip path uses the wrapper's vllm; docker path
Expand Down Expand Up @@ -470,7 +476,7 @@ piper-tts-server (services/piper-tts/)

| Server | Package | Command | Default port | Model | Backend |
|---|---|---|---|---|---|
| `services/vlm-server/` | `vlm-server` | `vlm_server` | 8100 | Cosmos-Reason1-7B | vLLM (pip or docker) |
| `services/vlm-server/` | `vlm-server` | `vlm_server` | 8100 | Cosmos3 Nano Reasoner | vLLM (pip or docker) |
| `services/stt-server/` | `stt-server` | `stt_server` | 8103 | parakeet-tdt-0.6b-v3 | NeMo ASR in-process |
| `services/magpie-tts/` | `magpie-tts-server` | `magpie_tts_server` | 8104 | magpie_tts_multilingual_357m | NeMo TTS in-process |
| `services/piper-tts/` | `piper-tts-server` | `piper_tts_server` | 8105 | rhasspy/piper-voices (ONNX) | piper-tts in-process |
Expand Down Expand Up @@ -558,9 +564,10 @@ remain provided by the shared voice runtime. The sample has no direct
Worker calls stt-server (8103), vlm-server (8100), and piper-tts-server
(8105) over HTTP via `xr-ai-models` SDK — no model weights loaded
in-process. The `models_config` key selects a structured deployment profile:
`models.local.json` manages the default services, `models.hosted.json` uses an
external NVIDIA NIM VLM, and `models.omni.json` reuses Nemotron-Omni on port
8108. These profiles separate adapter behavior, endpoint readiness and
`models.local.json` manages the default Cosmos3 Nano Reasoner service,
`models.hosted.json` uses the hosted Cosmos3 Nano Reasoner NIM, and
`models.omni.json` reuses Nemotron-Omni on port 8108. These profiles separate
adapter behavior, endpoint readiness and
credentials, and launcher ownership. Voice-gate knobs are configured via
`yaml/voice_gate.yaml`.

Expand Down Expand Up @@ -628,7 +635,7 @@ updated in the same commit**.
| `vllm_backend` / `vllm_image` YAML keys | `services/{vlm-server,embedding-server,llama-nemotron-llm,nemotron3-nano-llm,nemotron-omni-llm}/<server>.yaml`, every per-profile copy in `agent-samples/`, `docs/ai-services.md` |
| Container name used by a vllm wrapper | `_CONTAINER_NAME` in the wrapper's `__main__.py`, `stop_persistent_servers` names in `agent-samples/model-servers/main.py` |
| vlm-server model class or supported architectures | `services/vlm-server/vlm_server.yaml` comments |
| vlm-server YAML config keys (`model`, `model_cache`, …) | `services/vlm-server/vlm_server.yaml`, `agent-samples/simple-vlm-example/vlm_server.yaml` |
| vlm-server YAML config keys (`model`, `model_cache`, `async_scheduling`, …) | `services/vlm-server/vlm_server.yaml`, every per-profile copy in `agent-samples/` |
| cloudxr-runtime YAML config keys | `agent-samples/xr-render-demo/yaml/cloudxr_runtime.yaml`, `docs/adding-cloudxr.md` |
| `utils/xr-ai-launcher/xr_ai_launcher/_cloudxr_env.py` API | xr-render-scene + oxr-mcp + `services/cloudxr-runtime/cloudxr_runtime/__main__.py` imports, `agent-samples/xr-render-demo/main.py` (native-profile gate), `docs/adding-cloudxr.md`, `docs/xr-render-demo.md` (client-type section) |
| scene service YAML config keys | `agent-samples/xr-render-demo/scene/scene_service.yaml`, orchestrator process declaration, `docs/xr-render-demo.md` |
Expand Down
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ uv run model_servers
```

GPU profiles are auto-detected (`dual_48G_ada` / `spark` / `96G_blackwell`).
On first run each model downloads from HuggingFace (~50 GB total; can take
On first run each model downloads from HuggingFace (tens of GB; can take
tens of minutes). On subsequent runs the containers restart in under a minute.

The default `--vlm-llm-stack` starts Nemotron-3 Nano (8107), Cosmos (8100),
Expand All @@ -176,7 +176,7 @@ they cannot be stopped, avoiding GPU overcommit.
uv run model_servers --omni-stack
```

`HF_TOKEN` is required by default: without it the ~50 GB first-run download
`HF_TOKEN` is required by default: without it the large first-run download
can stall indefinitely. See [`docs/credentials.md`](docs/credentials.md)
for how to set it, or pass `--allow-anonymous` to run without one.

Expand All @@ -201,7 +201,8 @@ MCP client is involved. See the
[sample README](agent-samples/simple-vlm-example/README.md) for the worker
layout and configuration boundaries.

Uses `nvidia/Cosmos-Reason1-7B` (NVIDIA Open Model License + Apache 2.0).
Uses the text-output Reasoner from `nvidia/Cosmos3-Nano` by default. Standard
vLLM serving loads only the Reasoner, not the separate Generator pipeline.

There are two ways to run it:

Expand All @@ -214,7 +215,7 @@ uv sync
uv run simple_vlm_example
```

On the very first run weights download from HuggingFace (~23 GB; can take
On the very first run weights download from HuggingFace (tens of GB; can take
several minutes). `HF_TOKEN` is required by default; pass
`--allow-anonymous` to run without one (see
[`docs/credentials.md`](docs/credentials.md)).
Expand Down Expand Up @@ -265,6 +266,9 @@ after a moment, and you hear the reply through your speakers.
**Local model** — override the model weights or GPU settings by editing
`vlm_server.yaml` in the sample directory.

To use Cosmos-Reason1 instead, set `model: nvidia/Cosmos-Reason1-7B` in that
file and select `cosmos_vlm` as the VLM adapter preset in `models.local.json`.

**Remote model** — copy `yaml/models.hosted.json`, point its VLM endpoint at
your server, and select it in the worker config:

Expand Down
13 changes: 5 additions & 8 deletions agent-mcp-servers/vlm-mcp/vlm_mcp_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,16 @@ host: 0.0.0.0
port: 8240

# Model configuration — identifies the VLM backend by logical name "vlm".
# The cosmos_vlm preset targets services/vlm-server/ (Cosmos-Reason1-7B
# via vLLM). Only base_url needs to be set here; the preset fills in the rest.
# The cosmos3_nano_reasoner preset targets services/vlm-server/. Only base_url
# needs to be set here; the preset fills in the rest.
models:
vlm:
kind: preset:cosmos_vlm
kind: preset:cosmos3_nano_reasoner
base_url: http://localhost:8100

# httpx timeout applied to each call to vlm-server.
vlm_request_timeout_s: 60.0

# Whether to let the VLM produce chain-of-thought reasoning before answering.
# false (default) — suppresses <think> generation entirely; faster responses,
# better for real-time captioning and voice loops.
# true — enables full reasoning; more accurate on complex queries
# but adds several seconds of latency per request.
# Forward an enable_thinking chat-template flag for backends that support it.
# Cosmos3 Nano Reasoner does not use this Qwen-specific switch.
enable_thinking: false
8 changes: 4 additions & 4 deletions agent-mcp-servers/vlm-mcp/vlm_mcp_server/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
port: 8240
models:
vlm:
kind: preset:cosmos_vlm
kind: preset:cosmos3_nano_reasoner
base_url: http://localhost:8100
vlm_request_timeout_s: 60.0
enable_thinking: false
Expand Down Expand Up @@ -100,8 +100,8 @@ def _make_vlm_from_cfg(cfg: dict[str, Any]) -> tuple[VLMService, float]:
if nested:
vlm_entry["endpoint"] = target

# The cosmos_vlm preset defaults enable_thinking to False; an explicit
# top-level true must reach the wire by overriding default_extras.
# An explicit top-level true must reach backends that support the
# enable_thinking chat-template flag.
if enable_thinking:
target = dict(vlm_entry.get("adapter") or {}) if nested else vlm_entry
extras = dict(target.get("default_extras") or {})
Expand All @@ -119,7 +119,7 @@ def _make_vlm_from_cfg(cfg: dict[str, Any]) -> tuple[VLMService, float]:
elif vlm_server:
logger.warning(
"vlm_mcp_server.yaml: 'vlm_server' key is deprecated — "
"migrate to a 'models:' block with kind: preset:cosmos_vlm"
"migrate to a 'models:' block with an explicit VLM preset"
)
chat_template_kwargs: dict[str, Any] = {"enable_thinking": enable_thinking}
config = load_models_config_from_dict({
Expand Down
2 changes: 1 addition & 1 deletion agent-samples/model-servers/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
default / --vlm-llm-stack
stt — nvidia/parakeet-tdt-0.6b-v3 port 8103 (NeMo ASR)
agent-llm — NVIDIA-Nemotron-3-Nano-30B-A3B port 8107 (vLLM)
vlm — nvidia/Cosmos-Reason1-7B port 8100 (vLLM)
vlm — nvidia/Cosmos3-Nano Reasoner port 8100 (vLLM)
embedding — nvidia/llama-nemotron-embed-1b-v2 port 8109 (vLLM)

--omni-stack
Expand Down
10 changes: 5 additions & 5 deletions agent-samples/model-servers/yaml/96G_blackwell/vlm_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,22 @@
#
# vlm is SECOND in load order: agent-llm → vlm.
# free_at_startup ≈ 96 − 34 (agent-llm) − 2 (STT) ≈ 60 GiB.
# 0.20 × 96 = 19.2 GiB cap: ~16 GiB Cosmos-7B weights + ~3 GiB KV.
# 0.20 × 96 = 19.2 GiB cap: ~16 GiB Cosmos3 Reasoner weights + ~3 GiB KV.

model: nvidia/Cosmos-Reason1-7B
model: nvidia/Cosmos3-Nano
hf_token: ""
port: 8100
host: "0.0.0.0"
model_cache: ../../../../models
gpu_memory_utilization: 0.20
max_num_seqs: 4
enforce_eager: true
async_scheduling: true
max_videos_per_prompt: 0
# 4 fits within the default max_model_len=8192 (each Qwen2.5-VL image is
# ~1.5k tokens). Raise both together if multi-image turns get larger.
# Raise max_model_len too if multi-image turns outgrow the context window.
max_images_per_prompt: 4

# vLLM runtime: pip (this wrapper's venv) or docker (NGC nvcr.io/nvidia/vllm).
# See services/vlm-server/vlm_server.yaml for the full toggle docs.
vllm_backend: docker
vllm_image: nvcr.io/nvidia/vllm:26.04-py3
vllm_image: nvcr.io/nvidia/vllm:26.07-py3
22 changes: 11 additions & 11 deletions agent-samples/model-servers/yaml/dual_48G_ada/vlm_server.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# VLM server configuration — nvidia/Cosmos-Reason1-7B.
# VLM server configuration — Cosmos3 Nano Reasoner.
# Auto-discovered by xr-ai-launcher and passed as --config to vlm_server.
# OpenAI-compatible API at http://localhost:8100/v1 (accepts image_url in messages)

# HuggingFace model ID — any Qwen2.5-VL-compatible model is supported.
# nvidia/Cosmos-Reason1-7B — NVIDIA VLM; Apache 2.0 + NVIDIA Open Model License;
# no HF login required; ~16 GB VRAM at BF16
# nvidia/Cosmos-Reason2-8B — stronger reasoning; gated — requires HF login + license
model: nvidia/Cosmos-Reason1-7B
# HuggingFace model ID. Standard vLLM serving of nvidia/Cosmos3-Nano loads the
# Reasoner only; the Generator requires a separate `--omni` serving path.
# Cosmos-Reason1 compatibility uses nvidia/Cosmos-Reason1-7B with the
# cosmos_vlm client preset.
model: nvidia/Cosmos3-Nano

# HuggingFace token — required for gated models (e.g. Cosmos-Reason2-8B).
# Accept license at https://huggingface.co/nvidia/Cosmos-Reason2-8B first.
# HuggingFace token — optional for this public model and required for gated
# alternatives.
# !! Do not commit this file with a real token !!
hf_token: ""

Expand All @@ -26,12 +26,12 @@ model_cache: ../../../../models
# Single-image queries have short context so the KV budget is adequate.
gpu_memory_utilization: 0.43
enforce_eager: true
async_scheduling: true
max_videos_per_prompt: 0
# 4 fits within the default max_model_len=8192 (each Qwen2.5-VL image is
# ~1.5k tokens). Raise both together if multi-image turns get larger.
# Raise max_model_len too if multi-image turns outgrow the context window.
max_images_per_prompt: 4

# vLLM runtime: pip (this wrapper's venv) or docker (NGC nvcr.io/nvidia/vllm).
# See services/vlm-server/vlm_server.yaml for the full toggle docs.
vllm_backend: docker
vllm_image: nvcr.io/nvidia/vllm:26.04-py3
vllm_image: nvcr.io/nvidia/vllm:26.07-py3
10 changes: 5 additions & 5 deletions agent-samples/model-servers/yaml/spark/vlm_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,22 @@
#
# vlm is SECOND in load order: agent-llm → vlm.
# free_at_startup ≈ 120 − 42 (agent-llm) − 2 (STT) ≈ 76 GiB.
# 0.20 × 120 = 24 GiB cap: ~16 GiB Cosmos-7B weights + ~8 GiB KV.
# 0.20 × 120 = 24 GiB cap: ~16 GiB Cosmos3 Reasoner weights + ~8 GiB KV.

model: nvidia/Cosmos-Reason1-7B
model: nvidia/Cosmos3-Nano
hf_token: ""
port: 8100
host: "0.0.0.0"
model_cache: ../../../../models
gpu_memory_utilization: 0.20
max_num_seqs: 4
enforce_eager: true
async_scheduling: true
max_videos_per_prompt: 0
# 4 fits within the default max_model_len=8192 (each Qwen2.5-VL image is
# ~1.5k tokens). Raise both together if multi-image turns get larger.
# Raise max_model_len too if multi-image turns outgrow the context window.
max_images_per_prompt: 4

# vLLM runtime: pip (this wrapper's venv) or docker (NGC nvcr.io/nvidia/vllm).
# See services/vlm-server/vlm_server.yaml for the full toggle docs.
vllm_backend: docker
vllm_image: nvcr.io/nvidia/vllm:26.04-py3
vllm_image: nvcr.io/nvidia/vllm:26.07-py3
2 changes: 1 addition & 1 deletion agent-samples/simple-vlm-example/yaml/models.hosted.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"category": "vlm",
"adapter": {
"kind": "openai_compat",
"model_name": "nvidia/nemotron-nano-12b-v2-vl",
"model_name": "nvidia/cosmos3-nano-reasoner",
"capabilities": {"streaming": true, "vision": true}
},
"endpoint": {
Expand Down
2 changes: 1 addition & 1 deletion agent-samples/simple-vlm-example/yaml/models.local.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"deployment": {"ownership": "managed", "service": "stt"}
},
"vlm": {
"adapter": {"preset": "cosmos_vlm"},
"adapter": {"preset": "cosmos3_nano_reasoner"},
"endpoint": {"base_url": "http://localhost:8100", "readiness": "health"},
"deployment": {"ownership": "managed", "service": "vlm"}
},
Expand Down
5 changes: 3 additions & 2 deletions agent-samples/simple-vlm-example/yaml/vlm_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@
# vlm-server configuration for vlm-agent.
# Auto-discovered by the launcher and passed as --config to vlm_server.

model: nvidia/Cosmos-Reason1-7B
model: nvidia/Cosmos3-Nano
port: 8100
model_cache: ../../../models
async_scheduling: true

# vLLM runtime: pip (this wrapper's venv) or docker (NGC nvcr.io/nvidia/vllm).
# See services/vlm-server/vlm_server.yaml for the full toggle docs.
vllm_backend: docker
vllm_image: nvcr.io/nvidia/vllm:26.04-py3
vllm_image: nvcr.io/nvidia/vllm:26.07-py3
10 changes: 4 additions & 6 deletions agent-samples/xr-render-demo/yaml/models.nim.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,14 @@ agent_llm:
tool_calls: true
reasoning: true

# Visual question answering used by native vision functions.
# nvidia/cosmos-reason1-7b mirrors the local default (services/vlm-server
# also runs Cosmos-Reason1-7B), so the render-demo's ask_image prompts behave
# the same on NIM as locally. Swap to any hosted vision model on
# build.nvidia.com if you prefer.
# Visual question answering used by native vision functions. The hosted model
# is the Cosmos3 Nano Reasoner, matching the role of the local Reasoner-only
# vLLM service; it is not the separate Cosmos3 Generator.
vlm:
kind: openai_compat
category: vlm
base_url: https://integrate.api.nvidia.com
model_name: nvidia/cosmos-reason1-7b
model_name: nvidia/cosmos3-nano-reasoner
api_key_env: NGC_API_KEY
health_check: false
capabilities:
Expand Down
2 changes: 1 addition & 1 deletion agent-samples/xr-render-demo/yaml/models.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ tts:
base_url: http://localhost:8105

vlm:
kind: preset:cosmos_vlm
kind: preset:cosmos3_nano_reasoner
base_url: http://localhost:8100
5 changes: 3 additions & 2 deletions agent-sdk/xr-ai-models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ Built-in presets — see `xr_ai_models/presets/`:

| Preset | Service it targets | Notes |
|---|---|---|
| `cosmos_vlm` | vlm-server | image + video; `enable_thinking=false` by default. Video requires vlm-server's `max_videos_per_prompt >= 1` |
| `cosmos3_nano_reasoner` | vlm-server | default; Cosmos3 Nano text-output Reasoner, image + video; video requires `max_videos_per_prompt >= 1` |
| `cosmos_vlm` | vlm-server | Cosmos-Reason1 compatibility option; image + video; `enable_thinking=false` by default; video requires `max_videos_per_prompt >= 1` |
| `llama_nemotron` | llama-nemotron-llm-server | OpenAI tool calling via llama3_json (server-side) |
| `nemotron3_nano` | nemotron3-nano-llm-server | reasoning field: `reasoning` |
| `nemotron_omni` | nemotron-omni-llm-server | reasoning field: `reasoning_content`, vision + video |
Expand Down Expand Up @@ -186,7 +187,7 @@ are a profile change:
"category": "vlm",
"adapter": {
"kind": "openai_compat",
"model_name": "nvidia/cosmos-reason1-7b"
"model_name": "nvidia/cosmos3-nano-reasoner"
},
"endpoint": {
"base_url": "https://integrate.api.nvidia.com",
Expand Down
2 changes: 2 additions & 0 deletions agent-sdk/xr-ai-models/xr_ai_models/presets/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from copy import deepcopy
from typing import Any

from .cosmos3_nano_reasoner import COSMOS3_NANO_REASONER
from .cosmos_vlm import COSMOS_VLM
from .llama_nemotron import LLAMA_NEMOTRON
from .magpie_tts import MAGPIE_TTS
Expand All @@ -24,6 +25,7 @@


_PRESETS: dict[str, dict[str, Any]] = {
"cosmos3_nano_reasoner": COSMOS3_NANO_REASONER,
"cosmos_vlm": COSMOS_VLM,
"llama_nemotron": LLAMA_NEMOTRON,
"magpie_tts": MAGPIE_TTS,
Expand Down
Loading
Loading