Repository navigation
[test only] PR #92 + #96 on upstream PR #17862 (vLLM v0.33.0) - do not merge - #100
Draft
aung-san-i wants to merge 61 commits into
Draft
aung-san-i wants to merge 61 commits into
aung-san-i wants to merge 61 commits into
Conversation
… tests Implements section 2 of .ci/test/dynamo-heterog/XPU_CI_Test_Plan.md. New heterogeneous matrix legs (xpu-heterogeneous-run.yml, 7 -> 13 legs): - agg-x (INT-1): one aggregated XPU worker, the baseline - agg-router-2x (INT-11): two aggregated XPU workers behind the KV router - px2-dn-router (INT-12): 2 XPU prefill + NVIDIA decode behind the KV router - px-dn-8b (INT-6): px-dn on Qwen/Qwen3-8B - ex-pdx (EXT-15): XPU encode + XPU prefill-decode on one host, Qwen3-VL-2B - omni-ar-dit-1node-xpu (EXT-19): GLM-Image AR and DiT both on XPU Re-enabled pytest coverage (EXT-12/13), skipped since ai-dynamo#15406: - qwen3-vl-2b agg and agg_router profiles in vllm_xpu.py - xpu_1 marker on test_vllm_mm_router_e2e.py Issue ai-dynamo#15417 is still open; these need a run on the v0.31.0 XPU base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y omni leg px2-dn-router: set UCX_IB_GID_INDEX=1 on Prefill and Decode. The prefill replicas run on SR-IOV VFs that keep only their link-local GIDs; without a pinned index decode's UCX endpoint create failed with "Address not valid" and every request errored (run 41087132745). Same fix as omni_ar-dit_2nodes. omni-ar-dit-1node-xpu: removed from the matrix (13 -> 12 legs). GLM-Image's AR stage took 18.88 GiB on one 24 GB B60 and left -2.68 GiB for KV cache. It needs AR at TP=2, i.e. three B60s, more than intel01-b60 has. The manifest is kept, with the measured result in its header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The XPU workers run as root and create Qwen3-VL-2B's folder in the shared HF cache root-owned. huggingface_hub writes a lock file there even on a cache hit, so the bench pod failed to load the tokenizer with "Permission denied" (run 41127269054), after the encode -> prefill-decode request itself had succeeded. runAsUser: 0 matches the workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-captures the amd64 licence baseline in the same commit as the tag bump. The licences stage subtracts on (name, version), so bumping the tag alone leaves every moved package unsubtracted and fails the policy gate in compliance.Dockerfile before any test can run. Image digest: sha256:be3a9aea234be64ffc2816ac561c597e0f0b1f44a61a7f4912780e4d77450086 Baseline stem: vllm-openai-xpu@be3a9aea Opened by .github/workflows/xpu-base-image-watch.yaml Signed-off-by: dynamo-ops <170655669+dynamo-ops@users.noreply.github.com>
Move the CUDA runtime image and the ai-dynamo[vllm] extra to vLLM 0.33.0; CPU and XPU keep 0.32.0. vLLM-Omni stays at 0.32.0rc1 until a release compatible with core 0.33 is published. - Import FrontendArgs from its 0.33 module, keeping the 0.32 fallback. - Keep legacy KV block hashes for workers that publish KV events; 0.33 enables EAGLE/MTP lookahead hashes by default. - Update contract tests for the new BlockStored and EngineCoreOutput fields and the prompt-embeds width check. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…LLM v0.33.0 vLLM v0.33.0's NIXL connector refuses NIXL < 1.5.0, and the XPU image inherited the framework-level nixl_ref v1.4.1, so every disaggregated XPU worker exited at engine start (run 41492764713: px-dn, pn-dx, px-dn-hnd-tp2, px-dx). vLLM-Omni 0.31.0rc1 imports get_engine_zmq_addresses, which vLLM 0.33 renamed, so the omni DiT worker failed on import (omni-ar-dit-2nodes). No Omni release targets vLLM 0.33 yet; 0.32.0 has the renamed import. Transformers 5.14.1 / tokenizers 0.22.2 are below vLLM 0.33's floors (>=5.16.1, >=0.23.2). 5.19.0 is the newest release inside both vLLM 0.33's range (<5.21) and Omni 0.32's (<5.20). All three pins are scoped to vllm.xpu; CUDA and CPU are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Aung San aka Mike <41126828+aung-san-i@users.noreply.github.com>
The merge of main dropped vllm.xpu.nixl_ucx_ref, so the XPU image fell back to the dynamo-wide nixl_ucx_ref v1.22.0. main builds XPU with UCX v1.23.0-rc1 next to NIXL 1.5.0 (args.Dockerfile reads the per-device value first), so keep that pairing here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….33+ The vLLM sidecar now defaults structural tags on when vLLM reports version 0.33 or later, as the in-process vLLM worker and the SGLang sidecar (ai-dynamo#16494) already do. vLLM 0.33.0 is the first release whose gRPC server compiles these tags with xgrammar (vllm-project/vllm#61892); older servers send them to guidance, which rejects them, so the default stays off there. The version comes from ServerInfo.engine_version and is compared with pep440_rs. Folded in from ai-dynamo#17866. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
No vLLM-Omni release supports core vLLM 0.33 yet, and Omni 0.32 cannot start on it. Build the CUDA image's Omni from the head of vllm-omni#9648 (fe886bbb) through GitHub's source archive, under version 0.33.0.dev0, until Omni publishes a 0.33 release. Pin Transformers 5.19.0 and tokenizers 0.23.3 on CUDA, because Omni still caps Transformers below 5.20. Omni 0.33 appends stage_payload and stage_work to EngineCoreOutput; the renderer contract test expects them. CPU and XPU keep Omni 0.32.0rc1. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Workers that publish KV events default disable_lookahead_block_hashes to true. Document the default, its prefix-cache cost, and that turning it off on those workers is not supported. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
vLLM 0.33 derives an EAGLE/MTP speculative config from a speculators-format checkpoint inside create_engine_config, even with no --speculative-config or --spec-method. The lookahead-hash default returned before that, so such a worker publishing KV events kept lookahead hashes. Ask vLLM's own speculators detection first. Also move the KV-events test's imports to module scope. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…ry (ai-dynamo#17688) Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
The vLLM sidecar's new pep440_rs dependency was recorded in the root Cargo.lock but not in lib/bindings/python/Cargo.lock, so the bindings' --locked builds failed. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com> Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>
…#17788) Signed-off-by: xianlubird <xianlubird@gmail.com>
…eplay (ai-dynamo#17856) Signed-off-by: Yongming Ding <yongmingd@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…i-dynamo#17332) Signed-off-by: Anna Tchernych <atchernych@nvidia.com> Co-authored-by: Sachal Malick <s@chal.ai> Co-authored-by: Jacob Ioffe <jioffe@nvidia.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com> Co-authored-by: Dmitry Tokarev <191198572+dmitry-tokarev-nv@users.noreply.github.com>
Signed-off-by: Julia Kiczka <jkiczka@nvidia.com>
…o#17864) Signed-off-by: jthomson04 <jwillthomson19@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alex-Wengg <hanweng9@gmail.com>
Automated refresh from the public Dynamo Google Calendar. Generated by .github/workflows/community-events-refresh.yml. Signed-off-by: dynamo-ops <170655669+dynamo-ops@users.noreply.github.com>
Signed-off-by: Sachal Malick <ssrivastavam@nvidia.com> Signed-off-by: Sachal Malick <s@chal.ai>
…-v0.33-on-upstream-17862
…tion2-tests' into test/xpu-v0.33-on-upstream-17862
….0' into test/xpu-v0.33-on-upstream-17862
Omni 0.32.0 cannot start on vLLM 0.33: it imports FrontendArgs from vllm.entrypoints.launchers.cli_args, which vLLM 0.33 moved to vllm.entrypoints.common.cli_args (omni-ar-dit-2nodes, run 41521590494). ai-dynamo/dynamo PR ai-dynamo#17862 (448b252) taught install_vllm_omni.sh to build Omni from a <version>+g<sha> ref for every device and points the CUDA image at vllm-omni#9648's head (fe886bbb), which has the new import. Use the same ref for XPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7660) Signed-off-by: roman <pulromans@gmail.com>
…ng buffers (ai-dynamo#17935) Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
Signed-off-by: Sachal Malick <ssrivastavam@nvidia.com> Signed-off-by: Sachal Malick <s@chal.ai>
…s no tools (ai-dynamo#17914) Signed-off-by: svc-glamr@nvidia.com <svc-glamr@nvidia.com>
… modes (ai-dynamo#17712) Signed-off-by: Karen Chung <karenc@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Signed-off-by: Dr. Stefan Schimanski <sschimanski@nvidia.com>
Signed-off-by: jain-ria <riajain@NVIDIA.com>
…ynamo#16203) Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
…efresh (ai-dynamo#17913) Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…i-dynamo#17290) Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Dillon Cullinan <dcullinan@nvidia.com>
Signed-off-by: Julien Mancuso <jmancuso@nvidia.com>
…ai-dynamo#17915) Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…i-dynamo#17926) Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com> # Conflicts: # docs/fern/components/releases.data.ts # docs/fern/pages/reference/general/compatibility.mdx # docs/fern/pages/reference/general/releases-machine-readable.mdx
Signed-off-by: Julien Darve <jdarve@NVIDIA.com> Fix CI ahead of code freeze
…al tags PEP 440 orders pre-release and dev versions before their release, so vLLM nightlies such as 0.33.0rc1.dev649+gc3fd4ff44 compared below 0.33 and left structural tags off. Compare only the release segment. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
This was referenced Nov 3, 2026
…7862' into test/xpu-v0.33-on-upstream-17862 # Conflicts: # container/compliance/base_sboms/manifest.json
…rmers Review feedback on this PR: keep vllm_omni_ref in the shared vllm: key, as the previous submission did, instead of an XPU-only override, and give CPU the same Transformers/tokenizers pins. - vllm.vllm_omni_ref (the fallback for every device without its own; CUDA keeps ai-dynamo#17862's override) moves from v0.32.0rc1 to vllm-omni#9648's head fe886bbb, and the XPU-only copy is dropped. XPU resolves to the same ref. - vllm.cpu gets transformers_version 5.19.0 / tokenizers_version 0.23.2: Omni fe886bbb caps Transformers <5.20, and vLLM 0.32 accepts both. - The XPU pin comment now names Omni fe886bbb instead of Omni 0.32.0. CPU stays on vllm/vllm-openai-cpu:v0.32.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 0d4c4a7)
This release is vLLM v0.33.0 on every device. CUDA (upstream ai-dynamo#17862) and XPU already are; CPU was still on v0.32.0, which cannot run the shared Omni ref (fe886bbb imports FrontendArgs from vllm.entrypoints.common.cli_args, new in vLLM 0.33). The Transformers 5.19.0 / tokenizers 0.23.2 pins stay: they sit inside vLLM 0.33's range and under Omni's <5.20 cap. CPU has no baseline SBOM, so nothing else changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit bc392ea)
vllm-project/vllm-omni#9648 merged and v0.33.0rc1 is published on PyPI, so the installer no longer builds Omni from a GitHub source archive. Also correct the structural-tag docs: 0.33 release candidates and development builds use the default since 570b8ab. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…-on-upstream-17862
Upstream ai-dynamo#17862 (0298c8a) moved CUDA to vLLM-Omni v0.33.0rc1 from PyPI now that vllm-omni#9648 merged, and dropped install_vllm_omni.sh's <version>+g<sha> source-archive support. The shared vllm_omni_ref (XPU and CPU) still named the 0.33.0.dev0+gfe886bbb commit, which the installer would now look for on PyPI, so move it to v0.33.0rc1 as well. v0.33.0rc1 still caps transformers <5.20, so the Transformers 5.19.0 / tokenizers 0.23.2 pins stay; their comments now name v0.33.0rc1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e3c07b5)
vLLM-Omni 0.33.0rc1 imports PyAV at startup through media_utils.py, which the CUDA image removes under the codec policy, so every Omni worker failed to start. The installer now makes that import lazy and fails the build if the patch stops applying or Omni's pipeline registry cannot load without PyAV. Video-with-audio output keeps Dynamo's explicit PyAV error. The same import registers an Omni KV-cache type, after which vLLM skips its built-in types outside the engine; the scheduler tests now register them first, as EngineCore does. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
vLLM-Omni 0.33.0rc1 registers its experimental Physis pipelines (vllm-omni#9154) eagerly. Every Omni process, including the stage-engine processes Omni spawns, then imports PyAV, which the CUDA image removes under the codec policy, and Qwen3-TTS requests fail when stage 0 uses FlashInfer. Making the PyAV import lazy fixed only the first. The installer now removes the six Physis lines from Omni's pipeline registry on CUDA, fails the build if they change, and checks that the registry loads without PyAV. This replaces the media_utils.py patch, so the output formatter check and scheduler-test fixture it needed are reverted. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…-on-upstream-17862
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test only. Do not merge. Close this PR once the results are in.
This branch runs this fork's XPU work on top of upstream ai-dynamo/dynamo PR ai-dynamo#17862 (
jdarve/vllm-bump-v0.33.0), which moves Dynamo to vLLM v0.33.0. The local branch tracks that upstream branch, so later updates to ai-dynamo#17862 come in by merge.The head commit is:
jdarve/vllm-bump-v0.33.0ateb592aa(PR build(vllm): bump core to v0.33.0 and Omni to v0.33.0rc1 ai-dynamo/dynamo#17862, 7 commits on upstream main4cd8f4e);mainatb992db1(the fork's own XPU workflows and manifests);ci/heterog-plan-section2-tests(PR ci(xpu-heterog): add six test-plan legs and re-enable Qwen3-VL-2B XPU tests #92) at589b5bf: the new test legs;deps/xpu-vllm-v0.33.0(PR chore(xpu): bump vllm-openai-xpu to v0.33.0 #96) atb90b4cc: XPU image on vLLM v0.33.0 with NIXL 1.5.0, UCX v1.23.0-rc1, Transformers 5.19.0 and tokenizers 0.23.2;fe886bbb(0.33.0.dev0+gfe886bbb…), the same ref build(vllm): bump core to v0.33.0 and Omni to v0.33.0rc1 ai-dynamo/dynamo#17862 uses for CUDA, through build(vllm): bump core to v0.33.0 and Omni to v0.33.0rc1 ai-dynamo/dynamo#17862'sinstall_vllm_omni.sh. Omni 0.32.0 failed on vLLM 0.33 withNo module named 'vllm.entrypoints.launchers.cli_args'.The "Files changed" tab compares against this fork's
main, so it also lists ai-dynamo#17862's commits and the upstream commits under it.🤖 Generated with Claude Code