fix: update readme discord link - #271
Merged
Merged
Conversation
rmccorm4
approved these changes
Mar 18, 2025
dmitry-tokarev-nv
approved these changes
Mar 18, 2025
dmitry-tokarev-nv
enabled auto-merge (squash)
March 18, 2025 20:56
saturley-hall
requested changes
Mar 18, 2025
rmccorm4
self-requested a review
March 18, 2025 21:14
rmccorm4
reviewed
Mar 18, 2025
nvda-mesharma
disabled auto-merge
March 18, 2025 21:27
kylehh
pushed a commit
to kylehh/dynamo
that referenced
this pull request
Apr 11, 2025
- Setup venv ``` uv venv source .venv/bin/activate uv pip install pip uv pip install sgl-kernel --force-reinstall --no-deps uv pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/ ``` - Build: `cargo build --release --features sglang` - Run single node (make sure you're in the venv): `./tio out=sglang ~/llm_models/my_model` - Run Deepseek multi-gpu / multi-node: Node 1: ``` tio in=http out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 0 --dist-init-addr 10.217.98.122:9876 ``` Node 2: ``` tio in=none out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 1 --dist-init-addr 10.217.98.122:9876 ```
kylehh
pushed a commit
to kylehh/dynamo
that referenced
this pull request
Apr 11, 2025
Co-authored-by: Dmitry Tokarev <dtokarev@nvidia.com> Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
KavinKrishnan
added a commit
that referenced
this pull request
May 21, 2026
…e/target ## What this is and why we want it Cold-starting a large MoE model in TRT-LLM is dominated by weight loading, not by engine init. On Kimi K2.5 (TP=8, 685 GB) we see ~15-20 minutes per worker to load weights from a `shared-model-cache` PVC, with all GPUs idle during the load. For DeepSeek-V3.2, GLM-5, and other ~1 TB MoE models the penalty is even worse, and it's paid every time a replica scales up, restarts, or migrates. That stall is the bottleneck for HPA-driven autoscale, multi-tenant serving, and any deployment where the model size exceeds local-disk read throughput. ModelExpress (MX) is the upstream P2P weight transfer system in `ai-dynamo/modelexpress`. It runs a small gRPC server + Redis sidecar in the cluster, advertises which workers hold which model versions, and lets cold-starting workers pull weights via NIXL RDMA directly from a peer that's already serving. In practice that turns a ~20 minute disk load into ~2 seconds per rank at 360-500 Gbps (validated on Kimi K2.5 TP=8, GCP GB200, 4x 400G RoCE). Two upstream pieces landed in the last six weeks to make this work end- to-end: - **TRT-LLM PR #13531** (merged 2026-05-06, ships in 1.3.0rc15+): adds native `MXCheckpointLoader` for `checkpoint_format="MX"` in TRT-LLM's PyTorch backend. Does the actual RDMA receive on the target side, publishes via `publish_model_params(model)` on the source side, and falls back to HF disk loading when MX is unavailable. - **ModelExpress PR #202 + #267** (merged): provides `MxClient`, `MxLiveWeightLoader`, `publish_from_worker`, and `MX_POOL_REG` allocation-based NIXL pool registration on the client side. This PR is the Dynamo-side glue that lets users opt in from the command line. It adds `--model-express-url` to the TRT-LLM backend; when set, the engine probes the MX server at startup and auto-detects whether to act as the source (no peers yet, load from disk and publish) or target (peers already serving, pull via RDMA). No `--mx-role source|target` flag — same DGD spec works for both, which is what unlocks the HPA-driven scale-up case (every replica uses identical config; the auto-detect handles which role each one plays). There's no Linear/JIRA ticket; the workstream is tracked in the companion PRs below. ## How auto-detect works 1. Engine calls `_has_existing_sources()` which probes the MX server via `list_sources()` 2. **Sources found** -> target mode: sets `checkpoint_format="MX"` so TRT-LLM's upstream `MXCheckpointLoader` does the NIXL RDMA receive 3. **No sources** -> source mode: sets `MODEL_EXPRESS_URL` env var, workers auto-publish via `publish_from_worker()` after disk load ## Changes * **components/src/dynamo/trtllm/backend_args.py**: Add `--model-express-url` CLI argument (env `MODEL_EXPRESS_URL`). * **components/src/dynamo/trtllm/engine.py**: Plumb `model_express_url` through `TensorRTLLMEngine` and `get_llm_engine()`; when set, configure `checkpoint_format="MX"` and seed `MODEL_EXPRESS_URL` env so the upstream `MXCheckpointLoader` and `modelexpress.publish_from_worker` take over. * **components/src/dynamo/trtllm/workers/llm_worker.py**: Pass `model_express_url` from backend args to engine constructor. Use `config.exclude_tools_when_tool_choice_none` directly (drop the defensive getattr/hasattr guard; the field is part of the DynamoRuntimeConfig contract; .ai/python-guidelines.md flags defensive access on known types). Construct `RequestHandlerConfig` directly (drop the inspect.signature filter; the inspect import was inside the function body and the silent kwarg filtering hid contract mismatches; .ai/python-guidelines.md requires fail-fast). * **container/{context.yaml, templates/args.Dockerfile, templates/trtllm_runtime.Dockerfile}**: Add `ENABLE_MODELEXPRESS_P2P` + `MODELEXPRESS_REF` build args; install modelexpress from git when enabled. The new `RUN` step uses the same `--mount=type=cache,target= /home/dynamo/.cache/uv,uid=1000,gid=0,mode=0775,sharing=shared` cache mount as the rest of the file. * **recipes/deepseek-v3.2/trtllm/disagg/dep8x2/deploy.yaml**: DeepSeek- V3.2 disaggregated TP=8 prefill + TP=8 decode recipe with `--model-express-url` wired into both components. * **recipes/deepseek-v3.2/README.md** (new): Documents the deviation from `recipes/CONTRIBUTING.md` standard structure - this recipe intentionally has no `model-cache/` since MX P2P bypasses local cache for fast scale-up. Points operators at sibling DeepSeek recipes if they need the standard model-cache flow. ## Validation Validated end-to-end on GCP GB200 (Kimi K2.5 TP=8, 2 nodes per component): 16 target ranks x 90.75 GB transferred at 365-509 Gbps, end-to-end disaggregated inference verified. Same workload on a single- replica HPA-aggregated deploy: first replica ~22 minutes from disk, subsequent HPA-driven replicas ~5 minutes via RDMA at 361-583 Gbps/rank. ## Companion PRs * TRT-LLM PR #13531: `MXCheckpointLoader` (merged 2026-05-06, NVIDIA/TensorRT-LLM#13531) * ModelExpress PR #202: `MxLiveWeightLoader`, `publish_from_worker` (merged, ai-dynamo/modelexpress#202) * ModelExpress PR #267: `MX_POOL_REG` allocation-based registration (merged, ai-dynamo/modelexpress#267) * ModelExpress PR #271 / #272: deployment examples + MPI worker log flush fix (open, in review, do not block this PR) Signed-off-by: Kavin Krishnan <kavink@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update README to have persistent discord invite link