refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) by aoshen02 · Pull Request #48 · vllm-project/vime

aoshen02 · 2026-05-27T03:16:55Z

Summary

Follow-up to #45. Centralises _weight_version bookkeeping inside the data-carrying RPC so the IPC and distributed paths share the same contract slime uses, and removes two dead methods that were left behind by the PR #18 → PR #22 transition.

Why

After #22 introduced the colocated CUDA IPC path, vime had three update_weights* entry points whose version bookkeeping was inconsistent:

Entry	Path	Records `_weight_version`?
`update_weights_from_distributed`	NCCL	✅ in-RPC
`update_weights` (called from `IPCWeightTransferEngine.trainer_send_weights`)	IPC	❌ — vLLM's `IPCWeightTransferUpdateInfo` schema has no `weight_version` field
`update_weights_from_tensor` (PR #18 legacy)	—	✅ in-RPC, but no callers

The IPC gap caused the Weight version mismatch! Engine: /root/models/<...>, Updater: N failure under --ci-test. #45 patched it by piggybacking weight_version onto finish_weight_update, which fixed the bug but coupled version bookkeeping to a lifecycle hook instead of the data RPC. Any future IPC-style entry would face the same "remember to also write _weight_version" trap that PR #22 originally fell into.

slime avoids this trap by design: both IPC and distributed paths call engine.update_weights_from_tensor.remote(..., weight_version=N) (see slime/backends/sglang_utils/sglang_engine.py:update_weights_from_tensor) — same RPC name, version travels with the data.

Changes

vllm_engine.update_weights_from_tensor rewritten as the IPC entry point. Signature (update_info: dict, weight_version: str | None, flush_cache: bool); payload carries vLLM's IPCWeightTransferUpdateInfo (names / dtype_names / shapes / ipc_handles). Records _weight_version only after the POST succeeds — same post-POST ordering as update_weights_from_distributed and the review note on fix(weight-sync): record IPC weight version on engines so ci_test passes #45.
Delete vllm_engine.update_weights — was the entry for vLLM upstream's IPCWeightTransferEngine.trainer_send_weights; no longer used after step 5.
Delete vllm_engine._run_vllm_weight_update — dead helper; only the old update_weights_from_tensor body called it.
Revert finish_weight_update to a stateless POST — undoes the weight_version kwarg added in fix(weight-sync): record IPC weight version on engines so ci_test passes #45 since the version now lives in the data RPC.
update_weight_from_tensor._send_hf_chunk_via_ipc stops using IPCWeightTransferEngine.trainer_send_weights. Both slot_size == 1 and slot_size > 1 branches now call engine.update_weights_from_tensor.remote(update_info=..., weight_version=str(self.weight_version)) directly. vime keeps using reduce_tensor from torch.multiprocessing.reductions for IPC handle creation (via _build_ipc_update_info_from_named_tensors); we only own the dispatch.

Why not just keep #45

#45 worked but the wire shape did not match slime, and the bookkeeping write lived on a lifecycle hook (finish_weight_update) rather than the data RPC. Anyone adding a future IPC-flavoured entry point would face the same trap that PR #22 fell into. Centralising the write inside the data RPC removes the trap.

Tests

RecordingVLLMEngine learns update_weights_from_tensor so engine RPC call recording stays complete.
Renamed test_trainer_send_weights_uses_single_llm_handle_per_rank → test_send_via_ipc_dispatches_update_weights_from_tensor_with_version. Asserts the new RPC name + kwargs and that finish_weight_update is stateless (no kwargs).
Added test_update_weights_from_tensor_posts_ipc_update_info_and_records_version: asserts ipc_handles get cloudpickle'd into ipc_handles_pickled, metadata fields pass through, _weight_version advances on POST success.
Added test_update_weights_from_tensor_does_not_advance_version_on_failure: POST failure must not advance _weight_version (otherwise a retry would skip the resync).

Pre-existing test failures in tests/unit/backends/vllm_utils/test_vllm_engine.py (_weight_transfer_http_timeout, _response_json_or_fallback, server_host) are unchanged from main: main HEAD shows 8 failed / 24 passed, this PR shows 8 failed / 26 passed (the two new tests). Those are scope for a separate cleanup.

Test plan

Unit: tests/unit/backends/vllm_utils/test_vllm_engine.py — 26 passed (was 24 on main), 8 pre-existing failures unchanged
Unit: tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py — all passing
e2e: rerun the colocate+IPC suite once CI picks it up (tests/test_qwen2.5_0.5B_short.py, tests/test_qwen2.5_0.5B_ppo_critic_only_short.py, tests/test_qwen3_4B_ppo.py)

gemini-code-assist

Code Review

This pull request refactors the weight update mechanism from Megatron to vLLM via CUDA IPC. It replaces the use of IPCWeightTransferEngine.trainer_send_weights with direct calls to engine.update_weights_from_tensor.remote(...), aligning the RPC name and weight_version semantics with slime's interface so that the version travels directly with the data payload. Additionally, VLLMEngine is updated to handle update_weights_from_tensor directly, including cloudpickling of ipc_handles and advancing the tracked weight version only upon a successful POST. Obsolete methods are removed, and unit tests are updated accordingly. The reviewer suggests adding a unit test to cover the coordinator path when slot_size > 1 (multi-GPU/TP setups) to prevent regressions.

gemini-code-assist · 2026-05-27T03:18:26Z

+    assert first_call.kwargs["weight_version"] == "1"
+    # finish_weight_update is a stateless bookend now — no kwargs
+    assert len(engine.finish_weight_update.calls) == 1
+    assert engine.finish_weight_update.calls[0].kwargs == {}


While test_send_via_ipc_dispatches_update_weights_from_tensor_with_version covers the slot_size <= 1 path, the coordinator path for slot_size > 1 (which performs the all_gather_object and _merge_ipc_update_infos logic) is currently not covered by any unit test. Consider adding a unit test to cover this path to prevent regressions in multi-GPU/TP setups.

aoshen02 · 2026-05-27T03:28:02Z

CI baseline check: confirmed the 5 e2e-test-plugin-contracts failures (test_megatron_argument_validation, test_plugin_path_loading_contracts, test_plugin_runtime_hook_contracts, test_plugin_rollout_contracts, test_plugin_generate_contracts) and the Run pre-commit failure are pre-existing on main — same failures reproduce locally on origin/main (HEAD 68ca256) without any of this PR's changes applied. Not introduced by this PR.

Concretely for test_plugin_runtime_hook_contracts.py::test_runtime_hook_callsite_is_stable[rollout_data_postprocess] — the contract expects self.rollout_data_postprocess(self.args) in slime/backends/megatron_utils/actor.py, but main's actor.py doesn't contain that string (callsite drift). This PR doesn't touch actor.py at all.

Local unit-test result: 29 passed (24 → 29 = +5 new tests for this PR), 8 pre-existing failed (same 8 failing on origin/main).

…version-with-data) Background ---------- After PR #22 introduced the colocated CUDA IPC path, vime ended up with three ``update_weights*`` RPC entry points whose ``_weight_version`` bookkeeping was inconsistent: - ``update_weights_from_distributed`` (NCCL path): writes ``_weight_version`` inside the RPC, version travels with data — slime-style. - ``update_weights`` (IPC path, called from ``IPCWeightTransferEngine.trainer_send_weights``): forwarded vLLM's ``IPCWeightTransferUpdateInfo`` to ``/update_weights`` over HTTP but never recorded ``_weight_version`` — vLLM's payload schema does not carry it. - ``update_weights_from_tensor`` (PR #18 legacy entry): kept a SGLang-ish ``serialized_named_tensors`` payload and wrote ``_weight_version``, but had no callers in main. The IPC gap was the root cause of #41-era's "Weight version mismatch! Engine: /root/models/<...>, Updater: N" failure on every colocated test with ``--ci-test`` (fixed in #45 by piggybacking ``weight_version`` onto ``finish_weight_update``). slime's design avoids this entirely: both IPC and distributed call ``engine.update_weights_from_tensor.remote(..., weight_version=N)`` — same RPC name across both repos, with version travelling alongside the data in the same RPC. This PR ------- Rewire vime's IPC path to match slime's interface: 1. ``vllm_engine.update_weights_from_tensor`` is now the IPC entry point. Signature ``(update_info: dict, weight_version: str | None, flush_cache)``; payload carries vLLM's ``IPCWeightTransferUpdateInfo`` (names / dtype_names / shapes / ipc_handles), the trainer constructs it with ``reduce_tensor`` from ``torch.multiprocessing.reductions``. Records ``_weight_version`` only after the POST succeeds — mirrors ``update_weights_from_distributed``'s post-POST ordering so a failed transfer never advances the engine's tracked version. 2. Delete ``vllm_engine.update_weights`` — was the vLLM ``IPCWeightTransferEngine.trainer_send_weights`` entry, no longer used after step 4 below. 3. Delete ``vllm_engine._run_vllm_weight_update`` — dead helper that only ``update_weights_from_tensor``'s old SGLang-ish path called. 4. Revert ``finish_weight_update`` to a stateless POST — ``_weight_version`` now lives in the data-carrying RPC, so the bookend no longer needs to piggyback a kwarg. (Undoes the kwarg added in #45.) 5. Replace ``IPCWeightTransferEngine.trainer_send_weights(...)`` calls in ``_send_hf_chunk_via_ipc`` with direct ``engine.update_weights_from_tensor.remote(update_info=..., weight_version=...)`` for both slot_size paths (slot_size==1 and slot_size>1). vime keeps reusing vLLM's ``reduce_tensor`` for IPC handle creation (via ``_build_ipc_update_info_from_named_tensors``) — only the dispatch is ours — so we don't fork the vLLM IPC protocol, just route through our own RPC surface. Why not just keep #45's piggyback? - #45 worked but coupled version bookkeeping to the lifecycle hook ``finish_weight_update`` instead of the data RPC. The wire shape doesn't match slime's, and a new IPC-style entry point added later would have to remember to also write ``_weight_version`` — exactly the trap PR #22 fell into. Centralising the write inside the data RPC removes the trap. Unit tests ---------- - ``RecordingVLLMEngine`` learns ``update_weights_from_tensor`` so engine RPC call recording stays complete. - Renamed ``test_trainer_send_weights_uses_single_llm_handle_per_rank`` -> ``test_send_via_ipc_dispatches_update_weights_from_tensor_with_version``, asserts the new RPC name + kwargs (``update_info``, ``weight_version``) and that ``finish_weight_update`` is now stateless (no kwargs). - Added ``test_update_weights_from_tensor_posts_ipc_update_info_and_records_version``: asserts ipc_handles get cloudpickle'd into ipc_handles_pickled, metadata fields pass through, ``_weight_version`` advances on POST success. - Added ``test_update_weights_from_tensor_does_not_advance_version_on_failure``: asserts POST failure does not advance ``_weight_version`` (matches the same post-POST ordering review note from #45). Pre-existing test failures in tests/unit/backends/vllm_utils/test_vllm_engine.py (``_weight_transfer_http_timeout``, ``_response_json_or_fallback``, ``server_host``) are unchanged from main — main has 8 failed / 24 passed, this PR has 8 failed / 26 passed (the two added tests). Not in scope here. Signed-off-by: aoshen02 <aoshen@inferact.ai>

Two regressions in the previous commit only fire when Megatron TP != rollout-num-gpus-per-engine (e.g. parallel-check sweeps Megatron TP=1 with rollout TP=2). Both surface via ``tests/test_qwen3_0.6B_parallel_check.py``. Bug 1: leader gating wrong reference group ------------------------------------------ ``connect_rollout_engines`` used:: if mpu.get_tensor_model_parallel_rank() == 0: self._ipc_engine_coordinator = True The intent was "TP rank 0 within the engine GPU slot", but ``mpu.get_tensor_model_parallel_rank()`` is the Megatron TP rank, not the engine-slot rank. When Megatron TP=1, every trainer rank sees ``tp_rank=0`` and becomes a coordinator. For slot_size > 1, both ranks in the slot then call ``start_weight_update`` → the second call explodes:: Worker failed with error 'start_weight_update called while a weight update is already active. Call finish_weight_update first.' Fix: gate on ``rank == start`` (lowest trainer rank in the engine GPU range). Unique per slot regardless of Megatron parallelism. Bug 2: gather group wrong scope ------------------------------- ``_send_hf_chunk_via_ipc`` used ``mpu.get_tensor_model_parallel_group()`` to all_gather IPC payloads from peers in the engine slot. Again Megatron TP group ≠ engine slot when their world sizes differ. The merge then only had the coordinator's own UUID; downstream workers reading a different physical GPU got:: ValueError: IPC handle not found for GPU UUID <peer>. Available UUIDs: ['<coordinator>'] Fix: build per-slot process groups in ``connect_rollout_engines`` collectively (every trainer rank calls ``dist.new_group(slot_ranks)`` for every engine slot, keeps the one it belongs to). Use that group instead of Megatron's TP group for the gather and the trailing barrier. Validation ---------- Ran ``tests/test_qwen3_0.6B_parallel_check.py`` on 8×H200 with ``--num-rollout 2``. The test sweeps tp_size ∈ {1, 2, 4, 8} × pp_size ∈ {1, 2, 4} × cp_size ∈ {1, 2, 4, 8} for num_gpus ∈ {8, 4, 2} — every leg also uses ``--rollout-num-gpus-per-engine 2``, so the Megatron-TP=1 cases now exercise the new slot-group path. Pre-fix: fails on the very first Megatron-TP=1 config with bug 1 above; after bug 1 is patched, the next iteration fails with bug 2. Post-fix: the entire ~2-hour sweep completes with "Job succeeded" on every leg. Other tests already exercising IPC at Megatron-TP=2 == rollout-TP=2 are unaffected by this fix (``rank == start`` is equivalent to ``tp_rank == 0`` when the slot fits a single Megatron TP group). The following also pass post-fix as a sanity check: ``test_qwen3_4B_ppo``, ``test_qwen3_4B_ppo_train_critic_only``, ``test_qwen3_4B_ppo_disaggregate``, ``test_mimo_7B_mtp_only_grad``, ``test_moonlight_16B_A3B``, ``test_quick_start_glm4_9B``, ``test_qwen2.5_0.5B_{short,async_short,debug_rollout_then_train, ppo_critic_only_short}``, ``test_qwen3.5_0.8B_gsm8k_{short,async_short}``. Signed-off-by: aoshen02 <aoshen@inferact.ai> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

aoshen02 · 2026-05-27T13:25:12Z

/gemini review

gemini-code-assist

Code Review

This pull request refactors the weight update mechanism for colocated vLLM engines to align with slime's RPC contract. It bypasses vLLM's trainer_send_weights in favor of directly invoking update_weights_from_tensor with explicit fields and tracking the weight_version directly within the data-carrying RPC. It also introduces per-slot process groups to handle mismatches between Megatron TP and rollout TP. A critical issue was identified in the process group creation where the gloo backend must be explicitly specified to support all_gather_object calls.

gemini-code-assist · 2026-05-27T13:26:52Z

+            slot_start = colocate_gpu_offsets[i]
+            slot_end = slot_start + colocate_gpu_counts[i]
+            slot_ranks = list(range(slot_start, slot_end))
+            grp = dist.new_group(ranks=slot_ranks)


When creating the per-slot process groups, the backend is not specified, which defaults to the default process group's backend (typically nccl for GPU training in Megatron). However, dist.all_gather_object is used later on this group in _send_hf_chunk_via_ipc. Since the NCCL backend does not support object gathering (all_gather_object), this will result in a RuntimeError at runtime. Specifying backend="gloo" explicitly ensures the subgroup supports Python object collectives.

Suggested change

grp = dist.new_group(ranks=slot_ranks)

grp = dist.new_group(ranks=slot_ranks, backend="gloo")

not a problem

knlnguyen1802 · 2026-05-27T14:00:11Z

+            slot_start = colocate_gpu_offsets[i]
+            slot_end = slot_start + colocate_gpu_counts[i]
+            slot_ranks = list(range(slot_start, slot_end))
+            grp = dist.new_group(ranks=slot_ranks)


The _ipc_slot_group is created here without destroy anywhere. There is potential of memory leak if connect_rollout_engines is called many times even though it's a rare case. Consider fix or add a note on this

knlnguyen1802 · 2026-05-27T14:01:12Z

        if flush_cache:
            self.flush_cache()


It looks like this is never call under current code, same issue with update_weight_from_distributed can fix it in other PR

We might need to refactor this, not sure is it OK to keep self.node_rank = 0

SamitHuang · 2026-05-27T14:58:34Z

I've addressed the review feedback:

Explicitly specified the gloo backend for the process group creation to support all_gather_object calls.
Added a unit test (test_send_via_ipc_dispatches_update_weights_from_tensor_coordinator_multi_gpu) to cover the coordinator path when slot_size > 1.

knlnguyen1802 · 2026-05-27T15:05:40Z

I've addressed the review feedback:

Explicitly specified the gloo backend for the process group creation to support all_gather_object calls.

Added a unit test (test_send_via_ipc_dispatches_update_weights_from_tensor_coordinator_multi_gpu) to cover the coordinator path when slot_size > 1.

@SamitHuang Please fix DCO

SamitHuang

lgtm overall

…gpu test Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Restore update_weight_from_tensor and its unit test to main so this PR only carries the actor offload wake/sleep stabilization needed for the observed torch_memory_saver crash path.

…readiness timeout Two follow-up fixes in run-qwen3-8B-opd.sh discovered during PR #18 + PR #48 verification on the r3 image: (1) Drop --enable-prompt-logprobs from the vllm serve invocation. The flag does not exist in vllm/entrypoints/openai/{completion,chat_completion}; the upstream protocol gates prompt_logprobs purely as a per-request SamplingParams field (vllm/entrypoints/openai/completion/protocol.py:91, 271-274). The explanatory comment added in 6527f58 was incorrect — vllm has no server-side gate. Update the comment to reflect that and reference --max-logprobs (default 20) as the only relevant server-side cap. (2) Cap the teacher-readiness until-loop with a 600s deadline plus a PID liveness check, and clean up the teacher on script exit via trap. Without the deadline, when 'vllm serve' itself crashed (OOM, bad model path, port conflict) the loop spun forever — observed wedging the smoke run ~36 minutes before someone noticed. Now we exit with the last 100 lines of the teacher log either when the teacher process is gone or after 10 minutes, and EXIT trap kills the lingering teacher. No behavioral change when the teacher launches cleanly.

… calls The two _apply_monkey_patch_torch_reductions() call sites in this file (trainer send + vLLM worker hijack) are no-ops on the IPC path here: 1. We route IPC handles by physical GPU UUID dict key (set on the trainer side at _build_ipc_update_info_from_named_tensors via _current_gpu_uuid() from torch.cuda.get_device_properties().uuid). The receiver looks up by its own UUID, independent of args[6]. 2. vLLM's IPCWeightTransferEngine.receive_weights unconditionally overwrites args[6] with the receiver's local device_index before calling rebuild_cuda_tensor. Whatever a torch reductions patch encodes into args[6] is therefore discarded. The patch was the historical mechanism (sglang upstream) for translating device indices across CUDA_VISIBLE_DEVICES boundaries by stuffing UUID strings into args[6]. Our UUID-keyed dict + vLLM's explicit device_index override accomplish the same thing without the global torch reductions mutation. Also expand the _build_ipc_update_info_from_named_tensors docstring to spell out the UUID-keyed routing contract so future readers don't have to chase this through git history. Side effect: hf_weight_iterator_direct.py also calls monkey_patch_torch_reductions() at module-collective time. That call site is similarly decorative (only NCCL broadcast / all_gather collectives run there, no cross-process pickling) but lives outside this file's scope and is not touched here. Tracked alongside #29.

…dead code Follow-up to 39bf899 (deleted _apply_monkey_patch_torch_reductions from update_weight_from_tensor.py). With that helper gone, two more references are now dead in the vime IPC weight-transfer path: 1. hf_weight_iterator_direct.py:48 called monkey_patch_torch_reductions() at the top of _get_megatron_full_params(). On vime this never has effect on the IPC handle path: _get_megatron_full_params only runs NCCL broadcast/all_gather collectives (no cross-process pickling), and the chunks it returns are subsequently sent via PR #48's UUID-keyed {gpu_uuid: reduce_tensor(weight)} dict that vLLM's receiver routes by physical UUID + explicit args[6] overwrite. The call survives in slime/ miles upstream because their downstream path pickles tensors through sglang's MultiprocessingSerializer.serialize (ForkingPickler → reduce_tensor), where the patched encoding/decoding does real work; PR #48 does not use that pipeline, so the call here was incidentally inherited rather than functionally required. 2. slime/backends/megatron_utils/sglang.py's monkey_patch_torch_reductions re-export + __all__ entry now have no remaining importers in vime. Remove them. Also (this commit, B): 3. vllm_engine.py's update_weights_from_tensor docstring referred to "closures injected by _apply_monkey_patch_torch_reductions" as the reason for cloudpickle. That helper is gone; the cloudpickle is still correct because reduce_tensor returns a (rebuild_fn, args) tuple where the rebuild_fn is a module-level callable that JSON can't serialise. Update the docstring to reflect the actual reason. Net behaviour: identical — the deletions remove dead code paths. The patch_torch shim itself is still importable for callers outside vime (none currently in this tree). Cross-references: - #29 — issue documenting the no-op stub - 39bf899 — prior commit that deleted the helper from update_weight_from_tensor.py

Resolves conflicts with PR #55 (8ffcabf, "Fix vLLM IPC weight transfer for MiMo MTP training") which merged into main while PR #48 was open. PR #55 overlaps PR #48's IPC sender in update_weight_from_tensor.py: 1. _build_ipc_update_info_from_named_tensors return type - PR #48 (39bf899 docstring): returned dict[str, list], explained UUID-keyed routing as the contract that replaces the need for a torch reductions monkey-patch. - PR #55: changed return to tuple[dict[str, list], list[torch.Tensor]], adding weight_refs liveness guard (producer storage must stay alive until receiver opens the IPC handle). - Resolution: keep PR #55's tuple return + weight_refs; merge both docstring paragraphs (the UUID-routing rationale and the liveness rationale are orthogonal and both correct). 2. _send_hf_chunk_via_ipc barrier + cleanup - PR #48 (0d435a0): barrier group changed from tp_group to slot_group as part of the slot-leader gating fix (Bug 1/2 series). - PR #55: added ``del weight_refs`` after the barrier to release sender storage. - Resolution: keep PR #48's slot_group + PR #55's del weight_refs. Both improvements are orthogonal. Also propagate PR #55's tuple return into the slot_size <= 1 fast path which PR #48 introduced after PR #55 was written; that branch was still spreading the bare dict (``**local_info``) which now needs to unpack ``local_info, weight_refs = _build_...`` and ``del weight_refs`` after the ray.get() RPC. And drop the ``with patch(...) ._apply_monkey_patch_torch_reductions`` line in the unit test ``_run_update`` helper — the helper was deleted by my earlier commit 39bf899 on this branch, so the patch context would raise ``AttributeError`` after the merge. Pre-existing failure on this branch: ``test_connect_marks_one_coordinator_per_engine_gpu_slot`` fails on PR #48 HEAD (0d435a0, fd12344, and now the merge result) but passes on main; the failure ("Default process group has not been initialized") is independent of this merge and was already broken before main was pulled in. Not addressed here; should be fixed in a separate commit on PR #48. Verified: ``pytest tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py --deselect ::test_connect_marks_one_coordinator_per_engine_gpu_slot`` → 6 passed.

- update_weight_from_tensor.py module docstring: rewrite from a pure vLLM perspective. Step (2) now spells out the merged {uuid_G0: handle, uuid_G1: handle, ...} dict + collective_rpc fan-out inside the vLLM server, and step (3) makes version-with-data atomicity explicit. Drop the "match slime's sglang_engine signature" framing. - vllm_engine.update_weights_from_tensor: collapse the long sglang-vs-vLLM compare block into a focused docstring describing what the POST does and why ipc_handles needs cloudpickle. Add a Chinese call-stack walkthrough (trainer → ★this method★ → server collective_rpc → per-TP worker receive_weights) and note that `node_rank != 0` is a dead branch since VLLMEngine pins node_rank=0 (see PR #48 review comment for the follow-up cleanup). - Hoist base64/cloudpickle imports to module scope so the hot path no longer pays the per-call import overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

CalvinXKY · 2026-05-28T13:18:14Z

LGTM

The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* tests + CI: complete sglang→vllm rename across tests/ and .github/ Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * on_policy_distillation: port from SGLang to vLLM /v1/completions Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * tests: drop duplicate smoke test updates from PR40 * test(update_weight_from_tensor): drop stale _apply_monkey_patch_torch_reductions patch The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):`` context in _run_update suppressed a helper call that PR #48 has since deleted from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime). After that PR lands the patched attribute won't exist and this line raises AttributeError. Remove it now so the test survives PR #48 merge. The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the fake sglang module is intentionally kept: on this branch the production code still imports it via ``from ..sglang import monkey_patch_torch_reductions`` (both update_weight_from_tensor._apply_monkey_patch_torch_reductions on PR #40's view of main, and hf_weight_iterator_direct.py at module level). Removing the stub here would break the test on PR #40 alone; it can be dropped in a follow-up once PR #48 finishes removing every import site. Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py`` all 6 pass with this change applied to gcl/pr18-tests-ci HEAD. * tests: drop duplicate top-level test_update_weight_from_tensor.py The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(vllm_config): use real get_model_url default endpoint /inference/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop on_policy_distillation.py from tests+CI PR (now owned by runtime PR #18) The OPD vLLM /v1/completions migration is a runtime change; it was folded into the core-runtime PR (#18). Restore this file to main here so the two PRs no longer overlap on it. #18 merges first, so this lands via #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop unit-test files now owned by runtime PR #18 test_vllm_config.py + the plugin_contracts tests are coupled to #18's runtime rename (they import vllm_config / vllm_rollout, which #18 creates). They live in #18; remove them here so the two PRs don't overlap. #18 merges first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: restore vLLM rollout args dropped during sglang→vllm rename The mechanical sglang→vllm rename dropped several rollout knobs instead of mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph capture caps, speculative decoding, expert parallel). Restore them using the mapping established by the converted production scripts on main (run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM AsyncEngineArgs: --sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N --sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c --sglang-ep-size N -> --vllm-enable-expert-parallel --sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}' Also: - glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs; --sglang-max-running-requests was 16) and split out cuda-graph capture. - fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*). - test_vllm_config: rename test_update_weights_default_true → test_update_weights_defaults_to_none (it asserts `is None`). Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head, moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy, disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer is driven by --prefill-num-servers + the --vllm-config prefill/decode topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(plugin_contracts): migrate from sglang_rollout to vllm_rollout The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(update_weight_from_tensor): drop stale slime…megatron_utils.sglang mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): use method=mtp (not eagle) in vllm speculative config The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ckpt): add --dist-ckpt-optim-fully-reshardable for PAO+offload save/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>

…version-with-data) (#48) * refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) Background ---------- After PR #22 introduced the colocated CUDA IPC path, vime ended up with three ``update_weights*`` RPC entry points whose ``_weight_version`` bookkeeping was inconsistent: - ``update_weights_from_distributed`` (NCCL path): writes ``_weight_version`` inside the RPC, version travels with data — slime-style. - ``update_weights`` (IPC path, called from ``IPCWeightTransferEngine.trainer_send_weights``): forwarded vLLM's ``IPCWeightTransferUpdateInfo`` to ``/update_weights`` over HTTP but never recorded ``_weight_version`` — vLLM's payload schema does not carry it. - ``update_weights_from_tensor`` (PR #18 legacy entry): kept a SGLang-ish ``serialized_named_tensors`` payload and wrote ``_weight_version``, but had no callers in main. The IPC gap was the root cause of #41-era's "Weight version mismatch! Engine: /root/models/<...>, Updater: N" failure on every colocated test with ``--ci-test`` (fixed in #45 by piggybacking ``weight_version`` onto ``finish_weight_update``). slime's design avoids this entirely: both IPC and distributed call ``engine.update_weights_from_tensor.remote(..., weight_version=N)`` — same RPC name across both repos, with version travelling alongside the data in the same RPC. This PR ------- Rewire vime's IPC path to match slime's interface: 1. ``vllm_engine.update_weights_from_tensor`` is now the IPC entry point. Signature ``(update_info: dict, weight_version: str | None, flush_cache)``; payload carries vLLM's ``IPCWeightTransferUpdateInfo`` (names / dtype_names / shapes / ipc_handles), the trainer constructs it with ``reduce_tensor`` from ``torch.multiprocessing.reductions``. Records ``_weight_version`` only after the POST succeeds — mirrors ``update_weights_from_distributed``'s post-POST ordering so a failed transfer never advances the engine's tracked version. 2. Delete ``vllm_engine.update_weights`` — was the vLLM ``IPCWeightTransferEngine.trainer_send_weights`` entry, no longer used after step 4 below. 3. Delete ``vllm_engine._run_vllm_weight_update`` — dead helper that only ``update_weights_from_tensor``'s old SGLang-ish path called. 4. Revert ``finish_weight_update`` to a stateless POST — ``_weight_version`` now lives in the data-carrying RPC, so the bookend no longer needs to piggyback a kwarg. (Undoes the kwarg added in #45.) 5. Replace ``IPCWeightTransferEngine.trainer_send_weights(...)`` calls in ``_send_hf_chunk_via_ipc`` with direct ``engine.update_weights_from_tensor.remote(update_info=..., weight_version=...)`` for both slot_size paths (slot_size==1 and slot_size>1). vime keeps reusing vLLM's ``reduce_tensor`` for IPC handle creation (via ``_build_ipc_update_info_from_named_tensors``) — only the dispatch is ours — so we don't fork the vLLM IPC protocol, just route through our own RPC surface. Why not just keep #45's piggyback? - #45 worked but coupled version bookkeeping to the lifecycle hook ``finish_weight_update`` instead of the data RPC. The wire shape doesn't match slime's, and a new IPC-style entry point added later would have to remember to also write ``_weight_version`` — exactly the trap PR #22 fell into. Centralising the write inside the data RPC removes the trap. Unit tests ---------- - ``RecordingVLLMEngine`` learns ``update_weights_from_tensor`` so engine RPC call recording stays complete. - Renamed ``test_trainer_send_weights_uses_single_llm_handle_per_rank`` -> ``test_send_via_ipc_dispatches_update_weights_from_tensor_with_version``, asserts the new RPC name + kwargs (``update_info``, ``weight_version``) and that ``finish_weight_update`` is now stateless (no kwargs). - Added ``test_update_weights_from_tensor_posts_ipc_update_info_and_records_version``: asserts ipc_handles get cloudpickle'd into ipc_handles_pickled, metadata fields pass through, ``_weight_version`` advances on POST success. - Added ``test_update_weights_from_tensor_does_not_advance_version_on_failure``: asserts POST failure does not advance ``_weight_version`` (matches the same post-POST ordering review note from #45). Pre-existing test failures in tests/unit/backends/vllm_utils/test_vllm_engine.py (``_weight_transfer_http_timeout``, ``_response_json_or_fallback``, ``server_host``) are unchanged from main — main has 8 failed / 24 passed, this PR has 8 failed / 26 passed (the two added tests). Not in scope here. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(weight-sync): correct IPC slot leader gating and gather group Two regressions in the previous commit only fire when Megatron TP != rollout-num-gpus-per-engine (e.g. parallel-check sweeps Megatron TP=1 with rollout TP=2). Both surface via ``tests/test_qwen3_0.6B_parallel_check.py``. Bug 1: leader gating wrong reference group ------------------------------------------ ``connect_rollout_engines`` used:: if mpu.get_tensor_model_parallel_rank() == 0: self._ipc_engine_coordinator = True The intent was "TP rank 0 within the engine GPU slot", but ``mpu.get_tensor_model_parallel_rank()`` is the Megatron TP rank, not the engine-slot rank. When Megatron TP=1, every trainer rank sees ``tp_rank=0`` and becomes a coordinator. For slot_size > 1, both ranks in the slot then call ``start_weight_update`` → the second call explodes:: Worker failed with error 'start_weight_update called while a weight update is already active. Call finish_weight_update first.' Fix: gate on ``rank == start`` (lowest trainer rank in the engine GPU range). Unique per slot regardless of Megatron parallelism. Bug 2: gather group wrong scope ------------------------------- ``_send_hf_chunk_via_ipc`` used ``mpu.get_tensor_model_parallel_group()`` to all_gather IPC payloads from peers in the engine slot. Again Megatron TP group ≠ engine slot when their world sizes differ. The merge then only had the coordinator's own UUID; downstream workers reading a different physical GPU got:: ValueError: IPC handle not found for GPU UUID <peer>. Available UUIDs: ['<coordinator>'] Fix: build per-slot process groups in ``connect_rollout_engines`` collectively (every trainer rank calls ``dist.new_group(slot_ranks)`` for every engine slot, keeps the one it belongs to). Use that group instead of Megatron's TP group for the gather and the trailing barrier. Validation ---------- Ran ``tests/test_qwen3_0.6B_parallel_check.py`` on 8×H200 with ``--num-rollout 2``. The test sweeps tp_size ∈ {1, 2, 4, 8} × pp_size ∈ {1, 2, 4} × cp_size ∈ {1, 2, 4, 8} for num_gpus ∈ {8, 4, 2} — every leg also uses ``--rollout-num-gpus-per-engine 2``, so the Megatron-TP=1 cases now exercise the new slot-group path. Pre-fix: fails on the very first Megatron-TP=1 config with bug 1 above; after bug 1 is patched, the next iteration fails with bug 2. Post-fix: the entire ~2-hour sweep completes with "Job succeeded" on every leg. Other tests already exercising IPC at Megatron-TP=2 == rollout-TP=2 are unaffected by this fix (``rank == start`` is equivalent to ``tp_rank == 0`` when the slot fits a single Megatron TP group). The following also pass post-fix as a sanity check: ``test_qwen3_4B_ppo``, ``test_qwen3_4B_ppo_train_critic_only``, ``test_qwen3_4B_ppo_disaggregate``, ``test_mimo_7B_mtp_only_grad``, ``test_moonlight_16B_A3B``, ``test_quick_start_glm4_9B``, ``test_qwen2.5_0.5B_{short,async_short,debug_rollout_then_train, ppo_critic_only_short}``, ``test_qwen3.5_0.8B_gsm8k_{short,async_short}``. Signed-off-by: aoshen02 <aoshen@inferact.ai> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(weight-sync): use gloo backend for ipc payload gather, add multi-gpu test Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(weight-sync): drop dead _apply_monkey_patch_torch_reductions calls The two _apply_monkey_patch_torch_reductions() call sites in this file (trainer send + vLLM worker hijack) are no-ops on the IPC path here: 1. We route IPC handles by physical GPU UUID dict key (set on the trainer side at _build_ipc_update_info_from_named_tensors via _current_gpu_uuid() from torch.cuda.get_device_properties().uuid). The receiver looks up by its own UUID, independent of args[6]. 2. vLLM's IPCWeightTransferEngine.receive_weights unconditionally overwrites args[6] with the receiver's local device_index before calling rebuild_cuda_tensor. Whatever a torch reductions patch encodes into args[6] is therefore discarded. The patch was the historical mechanism (sglang upstream) for translating device indices across CUDA_VISIBLE_DEVICES boundaries by stuffing UUID strings into args[6]. Our UUID-keyed dict + vLLM's explicit device_index override accomplish the same thing without the global torch reductions mutation. Also expand the _build_ipc_update_info_from_named_tensors docstring to spell out the UUID-keyed routing contract so future readers don't have to chase this through git history. Side effect: hf_weight_iterator_direct.py also calls monkey_patch_torch_reductions() at module-collective time. That call site is similarly decorative (only NCCL broadcast / all_gather collectives run there, no cross-process pickling) but lives outside this file's scope and is not touched here. Tracked alongside #29. * refactor(weight-sync): finish removing monkey_patch_torch_reductions dead code Follow-up to 39bf899 (deleted _apply_monkey_patch_torch_reductions from update_weight_from_tensor.py). With that helper gone, two more references are now dead in the vime IPC weight-transfer path: 1. hf_weight_iterator_direct.py:48 called monkey_patch_torch_reductions() at the top of _get_megatron_full_params(). On vime this never has effect on the IPC handle path: _get_megatron_full_params only runs NCCL broadcast/all_gather collectives (no cross-process pickling), and the chunks it returns are subsequently sent via PR #48's UUID-keyed {gpu_uuid: reduce_tensor(weight)} dict that vLLM's receiver routes by physical UUID + explicit args[6] overwrite. The call survives in slime/ miles upstream because their downstream path pickles tensors through sglang's MultiprocessingSerializer.serialize (ForkingPickler → reduce_tensor), where the patched encoding/decoding does real work; PR #48 does not use that pipeline, so the call here was incidentally inherited rather than functionally required. 2. slime/backends/megatron_utils/sglang.py's monkey_patch_torch_reductions re-export + __all__ entry now have no remaining importers in vime. Remove them. Also (this commit, B): 3. vllm_engine.py's update_weights_from_tensor docstring referred to "closures injected by _apply_monkey_patch_torch_reductions" as the reason for cloudpickle. That helper is gone; the cloudpickle is still correct because reduce_tensor returns a (rebuild_fn, args) tuple where the rebuild_fn is a module-level callable that JSON can't serialise. Update the docstring to reflect the actual reason. Net behaviour: identical — the deletions remove dead code paths. The patch_torch shim itself is still importable for callers outside vime (none currently in this tree). Cross-references: - #29 — issue documenting the no-op stub - 39bf899 — prior commit that deleted the helper from update_weight_from_tensor.py * docs(weight-sync): drop sglang refs + add IPC call-stack notes - update_weight_from_tensor.py module docstring: rewrite from a pure vLLM perspective. Step (2) now spells out the merged {uuid_G0: handle, uuid_G1: handle, ...} dict + collective_rpc fan-out inside the vLLM server, and step (3) makes version-with-data atomicity explicit. Drop the "match slime's sglang_engine signature" framing. - vllm_engine.update_weights_from_tensor: collapse the long sglang-vs-vLLM compare block into a focused docstring describing what the POST does and why ipc_handles needs cloudpickle. Add a Chinese call-stack walkthrough (trainer → ★this method★ → server collective_rpc → per-TP worker receive_weights) and note that `node_rank != 0` is a dead branch since VLLMEngine pins node_rank=0 (see PR #48 review comment for the follow-up cleanup). - Hoist base64/cloudpickle imports to module scope so the hot path no longer pays the per-call import overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>

* tests + CI: complete sglang→vllm rename across tests/ and .github/ Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * on_policy_distillation: port from SGLang to vLLM /v1/completions Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * tests: drop duplicate smoke test updates from PR40 * test(update_weight_from_tensor): drop stale _apply_monkey_patch_torch_reductions patch The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):`` context in _run_update suppressed a helper call that PR #48 has since deleted from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime). After that PR lands the patched attribute won't exist and this line raises AttributeError. Remove it now so the test survives PR #48 merge. The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the fake sglang module is intentionally kept: on this branch the production code still imports it via ``from ..sglang import monkey_patch_torch_reductions`` (both update_weight_from_tensor._apply_monkey_patch_torch_reductions on PR #40's view of main, and hf_weight_iterator_direct.py at module level). Removing the stub here would break the test on PR #40 alone; it can be dropped in a follow-up once PR #48 finishes removing every import site. Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py`` all 6 pass with this change applied to gcl/pr18-tests-ci HEAD. * tests: drop duplicate top-level test_update_weight_from_tensor.py The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(vllm_config): use real get_model_url default endpoint /inference/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop on_policy_distillation.py from tests+CI PR (now owned by runtime PR #18) The OPD vLLM /v1/completions migration is a runtime change; it was folded into the core-runtime PR (#18). Restore this file to main here so the two PRs no longer overlap on it. #18 merges first, so this lands via #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop unit-test files now owned by runtime PR #18 test_vllm_config.py + the plugin_contracts tests are coupled to #18's runtime rename (they import vllm_config / vllm_rollout, which #18 creates). They live in #18; remove them here so the two PRs don't overlap. #18 merges first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: restore vLLM rollout args dropped during sglang→vllm rename The mechanical sglang→vllm rename dropped several rollout knobs instead of mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph capture caps, speculative decoding, expert parallel). Restore them using the mapping established by the converted production scripts on main (run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM AsyncEngineArgs: --sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N --sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c --sglang-ep-size N -> --vllm-enable-expert-parallel --sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}' Also: - glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs; --sglang-max-running-requests was 16) and split out cuda-graph capture. - fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*). - test_vllm_config: rename test_update_weights_default_true → test_update_weights_defaults_to_none (it asserts `is None`). Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head, moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy, disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer is driven by --prefill-num-servers + the --vllm-config prefill/decode topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(plugin_contracts): migrate from sglang_rollout to vllm_rollout The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(update_weight_from_tensor): drop stale slime…megatron_utils.sglang mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): use method=mtp (not eagle) in vllm speculative config The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ckpt): add --dist-ckpt-optim-fully-reshardable for PAO+offload save/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>

gemini-code-assist Bot reviewed May 27, 2026

View reviewed changes

aoshen02 force-pushed the aoshen/align-ipc-rpc-with-slime branch from 8bc7586 to aead629 Compare May 27, 2026 03:25

aoshen02 force-pushed the aoshen/align-ipc-rpc-with-slime branch from aead629 to 60a3b30 Compare May 27, 2026 04:12

aoshen02 force-pushed the aoshen/align-ipc-rpc-with-slime branch from 60a3b30 to 3f93600 Compare May 27, 2026 04:24

This was referenced May 27, 2026

tests + CI: complete sglang→vllm rename across tests/ and .github/ #40

Merged

examples/scripts: complete sglang→vllm rename + drop SGLang-only examples #39

Merged

gemini-code-assist Bot reviewed May 27, 2026

View reviewed changes

knlnguyen1802 reviewed May 27, 2026

View reviewed changes

SamitHuang approved these changes May 27, 2026

View reviewed changes

fix(weight-sync): use gloo backend for ipc payload gather, add multi-…

0d435a0

…gpu test Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>

SamitHuang force-pushed the aoshen/align-ipc-rpc-with-slime branch from 9df62d8 to 0d435a0 Compare May 27, 2026 15:07

CalvinXKY mentioned this pull request May 27, 2026

[Bugfix] stabilize actor offload wake/sleep flow (follow-up to #48) #53

Closed

5 tasks

This was referenced May 28, 2026

[Bug] vLLM IPC weight transfer sender still on 2-tuple protocol after vllm#37476; receiver expects bare 15-tuple #58

Closed

[BugFix]:Fix vLLM IPC weight transfer for MiMo MTP training #55

Merged

This was referenced May 28, 2026

[Bug] examples/on_policy_distillation: bad vllm CLI flag, unbounded teacher-readiness loop, and sglang-shape reward parser (fixed in PR #18) #59

Open

fix(examples/opd): drop fictional --enable-prompt-logprobs flag from teacher launch #62

Merged

aoshen02 mentioned this pull request May 28, 2026

test: drop stale _apply_monkey_patch_torch_reductions patch context #63

Merged

aoshen02 added 2 commits May 28, 2026 08:41

aoshen02 force-pushed the aoshen/align-ipc-rpc-with-slime branch from d4fcbf6 to d6423e8 Compare May 28, 2026 12:35

aoshen02 force-pushed the aoshen/align-ipc-rpc-with-slime branch from d6423e8 to fef1c3b Compare May 28, 2026 12:40

CalvinXKY merged commit 76ef423 into main May 28, 2026
10 of 13 checks passed

knlnguyen1802 mentioned this pull request May 29, 2026

[Bugfix] [weight-sync] align colocated IPC coordinator/gather with vLLM engine slot #57

Closed

2 tasks

aoshen02 deleted the aoshen/align-ipc-rpc-with-slime branch June 8, 2026 14:17

	grp = dist.new_group(ranks=slot_ranks)
	grp = dist.new_group(ranks=slot_ranks, backend="gloo")

Uh oh!

Conversation

aoshen02 commented May 27, 2026

Summary

Why

Changes

Why not just keep #45

Tests

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

aoshen02 commented May 27, 2026

Uh oh!

aoshen02 commented May 27, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 27, 2026

Choose a reason for hiding this comment

Uh oh!

aoshen02 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

knlnguyen1802 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

knlnguyen1802 May 27, 2026

Choose a reason for hiding this comment

Uh oh!

aoshen02 May 28, 2026

Choose a reason for hiding this comment

Uh oh!

SamitHuang commented May 27, 2026

Uh oh!

knlnguyen1802 commented May 27, 2026

Uh oh!

SamitHuang left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CalvinXKY commented May 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SamitHuang left a comment •

edited

Loading