[Clean] Remove SGlang runtime code by gcanlin · Pull Request #18 · vllm-project/vime

gcanlin · 2026-05-20T08:59:18Z

Close #7.

pre-commit run

(.venv) root@coder:~/vime-workspace/vime# pre-commit run
check yaml...............................................................Passed
check for case conflicts.................................................Passed
detect private key.......................................................Passed
check for added large files..............................................Passed
fix requirements.txt.....................................................Passed
ruff check...............................................................Passed
autoflake................................................................Passed
isort....................................................................Passed
Format code..............................................................Passed

gemini-code-assist

Code Review

This pull request migrates the rollout engine backend from SGLang to vLLM, updating configuration parsing, CLI arguments, and internal utilities to support vLLM while maintaining backward compatibility for legacy YAML configurations. The review identified a critical bug where engine overrides were not being applied, a feature regression regarding deterministic rollouts, and an unused parameter in the FP8 helper functions that should be removed.

gemini-code-assist · 2026-05-20T09:04:06Z

                base_gpu_id=base_gpu_id,
                model_path=self.model_path,
-                sglang_overrides=self.sglang_overrides,
+                sglang_overrides=self.engine_overrides,


The engine_overrides from the rollout-config.yml are passed to the VLLMEngine constructor as sglang_overrides, but VLLMEngine does not apply these overrides. The VLLMEngine's launch_server_process function builds command-line arguments from the main args namespace and does not incorporate these overrides. This is a critical bug as it means per-group configurations in the YAML file will be ignored.

To fix this, VLLMEngine needs to be modified to accept and apply these overrides when launching the vLLM server process, similar to how the old sglang_engine worked. The parameter should also be renamed from sglang_overrides to engine_overrides for consistency.

gemini-code-assist · 2026-05-20T09:04:06Z

+    for _idx, sample in enumerate(group):
        current_sampling_params = sampling_params.copy()
-        if getattr(args, "sglang_enable_deterministic_inference", False):
-            seed = state.group_sampling_seeds[idx]
-            current_sampling_params["seed"] = seed
        tasks.append(
            asyncio.create_task(generate_and_rm(args, sample, current_sampling_params, evaluation=evaluation))
        )


The logic for deterministic rollouts, previously controlled by sglang_enable_deterministic_inference, has been removed. This is a feature regression that affects reproducibility. vLLM supports deterministic sampling via the seed parameter.

This functionality should be restored. You could re-introduce logic to set a seed for each sample in a group. This would likely involve:

Adding a new argument, e.g., --enable-deterministic-rollout.

In GenerateState.__init__, re-adding logic to create group_sampling_seeds based on rollout_seed and n_samples_per_prompt when the new argument is enabled.

In generate_and_rm_group and eval_rollout_single_dataset, setting the seed in sampling_params for each sample.

Suggested change

for _idx, sample in enumerate(group):

current_sampling_params = sampling_params.copy()

if getattr(args, "sglang_enable_deterministic_inference", False):

seed = state.group_sampling_seeds[idx]

current_sampling_params["seed"] = seed

tasks.append(

asyncio.create_task(generate_and_rm(args, sample, current_sampling_params, evaluation=evaluation))

)

for idx, sample in enumerate(group):

current_sampling_params = sampling_params.copy()

if getattr(args, "enable_deterministic_rollout", False):

# NOTE: This requires re-adding group_sampling_seeds to GenerateState

# and a new --enable-deterministic-rollout argument.

seed = state.group_sampling_seeds[idx]

current_sampling_params["seed"] = seed

tasks.append(

asyncio.create_task(generate_and_rm(args, sample, current_sampling_params, evaluation=evaluation))

)

I actually think we should keep it, it's a good feature.

Agreed — restored as --rollout-enable-deterministic with vLLM-native seed plumbing in GenerateState, generate_and_rm_group, and eval_rollout_single_dataset.

gemini-code-assist · 2026-05-20T09:04:06Z

+        )
+        return out_w, out_s
+
+    def transform_scale_ue8m0(sf: torch.Tensor, mn: int, use_torch_impl: bool = False):


The use_torch_impl parameter is unused within the transform_scale_ue8m0 function. This appears to be dead code and should be removed to improve code clarity.

Suggested change

def transform_scale_ue8m0(sf: torch.Tensor, mn: int, use_torch_impl: bool = False):

def transform_scale_ue8m0(sf: torch.Tensor, mn: int):

gcanlin · 2026-05-22T01:53:11Z

I ran Qwen3-0.6B and get the picture below. The functionality is fine but I'm not sure why the reward is down. Is it for model size? cc @CalvinXKY @aoshen02

CalvinXKY · 2026-05-22T02:06:34Z

I ran Qwen3-0.6B and get the picture below. The functionality is fine but I'm not sure why the reward is down. Is it for model size? cc @CalvinXKY @aoshen02

Consider using a simpler dataset or a larger model.

CalvinXKY · 2026-05-22T06:41:54Z

-    "quant_weight_ue8m0",
-    "transform_scale_ue8m0",
-    "should_deepgemm_weight_requant_ue8m0",
-    "monkey_patch_torch_reductions",


The monkey_patch_torch_reductions is needed for update_weight_from_tensor.py. If deleted, we need to consider whether update_weight_from_tensor can be implemented properly.

Yeah. I ported it here: slime/backends/megatron_utils/update_weight/torch_patch.py. And didn't change the behavior in this PR. This patch should be dependent from SGLang.

aoshen02 · 2026-05-22T08:45:27Z

/gemini review this

gcanlin · 2026-05-22T08:47:52Z

+vLLM is the primary source; sglang is kept as a fallback so that
+in-flight deployments that still have sglang installed are not broken.
+All symbols fall back to ``None`` when neither backend is available,
+which disables the UE8M0 requantization path.


This file hasn't been tested. I don't have hopper to test FP8. So there still exists the fallback to sglang. When we supported this feature, we can remove it totally.

gemini-code-assist

Code Review

This pull request migrates the rollout engine from SGLang to vLLM, which includes renaming CLI arguments, configuration classes, and internal modules to reflect the change. It introduces several utility files to transition away from SGLang dependencies and updates the project requirements, examples, and tests. The review feedback highlights a logic error in the CUDA IPC device mapping fix, suggests more specific exception handling, recommends adhering to PEP 8 import standards, and identifies unused code and parameters that should be removed.

gemini-code-assist · 2026-05-22T08:48:03Z

+    def _rebuild_cuda_tensor_modified(*args):
+        return reductions._rebuild_cuda_tensor_original(*args)


The current implementation of _rebuild_cuda_tensor_modified is a no-op because it simply forwards all arguments (including the potentially incorrect device ID) to the original function. To fix the CUDA IPC device mapping issue described in the docstring, this function should override the device argument with the current process's GPU assignment.

Suggested change

def _rebuild_cuda_tensor_modified(*args):

return reductions._rebuild_cuda_tensor_original(*args)

def _rebuild_cuda_tensor_modified(device, *args):

# Force the tensor to the current process's device to fix IPC mapping

return reductions._rebuild_cuda_tensor_original(torch.cuda.current_device(), *args)

gemini-code-assist · 2026-05-22T08:48:03Z

+# ---------------------------------------------------------------------------
+# Primary: vLLM fp8 helpers
+# ---------------------------------------------------------------------------
+try:


Catching the base Exception class is generally discouraged as it can swallow unexpected errors such as KeyboardInterrupt or SystemExit. It is better to catch specific exceptions like ImportError or AttributeError when checking for backend availability.

gemini-code-assist · 2026-05-22T08:48:03Z

+        weight_dequant: torch.Tensor,
+        weight_block_size: list[int],
+    ):
+        assert weight_block_size == [128, 128]


The assertion weight_block_size == [128, 128] is strict on the container type. If a tuple (128, 128) is passed, the assertion will fail even if the values are correct. Converting the input to a list before comparison would make the check more robust.

Suggested change

assert weight_block_size == [128, 128]

assert list(weight_block_size) == [128, 128]

gemini-code-assist · 2026-05-22T08:48:03Z

+        flat = weight_dequant.view(-1, k)
+        out_w_flat, out_s_flat = _vllm_per_block_cast(flat, block_size=[128, 128], use_ue8m0=True)
+        out_w = out_w_flat.view(*batch_dims, n, k)
+        from math import ceil


According to PEP 8, imports should be at the top level of the module. Moving math.ceil to the top level (within a try-except block if necessary) improves readability and avoids the overhead of repeated imports during function calls.

gemini-code-assist · 2026-05-22T08:48:03Z

+        )
+        return out_w, out_s
+
+    def transform_scale_ue8m0(sf: torch.Tensor, mn: int, use_torch_impl: bool = False):


The parameter use_torch_impl is defined in the function signature but is not used within the function body. It should be removed if it is not intended to be implemented.

gemini-code-assist · 2026-05-22T08:48:03Z

+class _SafeUnpickler(io.BytesIO.__class__):
+    pass
+
+
+def _safe_load(data: bytes):
+    import pickle  # noqa: S403
+
+    return pickle.loads(data)  # noqa: S301


The _SafeUnpickler class is incorrectly defined (inheriting from io.BytesIO.__class__, which is type) and is not used anywhere in the module. Additionally, the _safe_load function is also unused. These should be removed to clean up the codebase.

CalvinXKY · 2026-05-22T08:50:34Z

@gcanlin
gsm8K works well:

The raw_reward is still unstable. Usually, the evaluation (eval) is taken as the standard.

aoshen02 · 2026-05-22T09:25:06Z

Seems like several places in this file still use sglang stuff?

aoshen02 · 2026-05-22T09:30:28Z

Thanks for the pr. So I have a general question that will we rename sglang to rollout instead of vllm directly, I've seen some are vllm while some are rollout.

aoshen02 · 2026-05-22T09:36:24Z

+class TestRolloutConfigLegacySglangKey:
+    def test_update_weights_default_none_with_sglang_key(self):
+        """Legacy 'sglang:' key is still parsed; update_weights defaults to None."""
+        from slime.utils.rollout_config import RolloutConfig


Do we need to have a legacy test file?

aoshen02 · 2026-05-22T09:45:34Z

Remaining `sglang` references after this PR

After grepping the branch (gcl/clean-sglang), there are still quite a few sglang strings left across the codebase. Listing them by file — are these intended to be cleaned up in a follow-up, or should they be part of this PR?

`slime/` core

File	Lines	Nature
`slime/backends/megatron_utils/fp8_helpers.py`	L3-4, L58, L61, L65	Comments + live `from sglang.srt...` fallback imports
`slime/backends/megatron_utils/megatron_to_hf/__init__.py`	L62	Comment
`slime/backends/megatron_utils/megatron_to_hf/gpt_oss.py`	L7	Docstring
`slime/backends/megatron_utils/update_weight/hf_weight_iterator_bridge.py`	L17	Comment
`slime/backends/megatron_utils/update_weight/tensor_bucket.py`	L3-4	"Ported from sglang.srt..." comments
`slime/backends/megatron_utils/update_weight/torch_patch.py`	L3-4	"Ported from sglang.srt..." comments
`slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py`	L83	Comment
`slime/backends/megatron_utils/update_weight/update_weight_from_tensor.py`	L163	Comment
`slime/backends/vllm_utils/arguments.py`	L3, L81, L100, L296	Comments / docstrings referencing `sglang_utils/`
`slime/backends/vllm_utils/vllm_engine.py`	L416, L425	Functional: parameter still named `sglang_overrides`
`slime/backends/vllm_utils/vllm_engine.py`	L19, L266, L342, L389, L517, L598, L614, L640, L696, L713, L732, L748, L782, L796, L815, L831, L844, L856	Docstring comparisons "SGLang: POST /..."
`slime/ray/actor_group.py`	L54	Comment
`slime/ray/rollout.py`	L145-152	Functional: 7 `SGLANG_*` env vars set at launch
`slime/ray/rollout.py`	L169	Functional: keyword arg `sglang_overrides=` passed to `VllmEngine`
`slime/rollout/vllm_rollout.py`	L504	Comment
`slime/utils/logging_utils.py`	L11	`# ref: SGLang`
`slime/utils/multiproc_utils.py`	L3-4	"Ported from sglang.srt..." comments
`slime/utils/rollout_config.py`	L3-4, L25, L153-160	Comments + functional: `elif "sglang" in data` backward-compat parsing

`tests/`

All test files below use a sglang_args variable and pass --sglang-* prefixed CLI flags:

tests/test_qwen2.5_0.5B_short.py
tests/test_qwen2.5_0.5B_async_short.py
tests/test_qwen2.5_0.5B_debug_rollout_then_train.py
tests/test_qwen2.5_0.5B_ppo_critic_only_short.py
tests/test_qwen3.5_0.8B_gsm8k_short.py
tests/test_qwen3.5_0.8B_gsm8k_async_short.py
tests/test_qwen3_0.6B_parallel_check.py
tests/test_qwen3_4B_ckpt.py
tests/test_qwen3_4B_ppo.py
tests/test_qwen3_4B_ppo_disaggregate.py
tests/test_qwen3_4B_ppo_train_critic_only.py
tests/test_qwen3_30B_A3B.py
tests/test_qwen3_30B_A3B_r3.py
tests/test_qwen3.6_35B_A3B_pd_mooncake.py (16 occurrences including --sglang-disaggregation-transfer-backend mooncake)
tests/test_moonlight_16B_A3B.py
tests/test_moonlight_16B_A3B_r3.py
tests/test_mimo_7B_mtp_only_grad.py
tests/test_quick_start_glm4_9B.py
tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_distributed.py — function name test_source_no_sglang_dist_broadcast_fallback
tests/utils/test_rollout_config.py — test method test_legacy_sglang_key_accepted
tests/utils/test_rollout_config_legacy.py — entire file is backward-compat tests for the sglang: YAML key

`train.py` / `train_async.py`

2 comments each

`tools/`

File	Detail
`tools/analyze_profile.py`	3 occurrences (description, flag hints)
`tools/convert_hf_to_torch_dist.py`	1 occurrence (help text)
`tools/convert_torch_dist_to_hf_parallel.py`	L276: `args.sglang_enable_ep_moe = False` (functional)
`tools/profile_rollout.py`	2 occurrences (description, default output path `/tmp/sglang_profile`)
`tools/replay_openai_jsonl.py`	1 occurrence (description)

`slime_plugins/`

rollout_buffer_example.py L180: args.sglang_router_ip / sglang_router_port (functional)
rollout_buffer_example.sh L4: pkill -9 sglang

Two questions before proceeding

SGLANG_* env vars in rollout.py — these are SGLang-internal runtime flags. Are they still needed (e.g. for a remaining SGLang fallback path), or can they be dropped entirely?
rollout_config.py backward-compat sglang: key — removing elif "sglang" in data will silently break existing YAML configs. Is a deprecation window intended, or is a hard break acceptable?

gcanlin · 2026-05-24T15:26:43Z

Thanks for the pr. So I have a general question that will we rename sglang to rollout instead of vllm directly, I've seen some are vllm while some are rollout.

I was thinking about it. Currently, this PR is making some general names as rollout and some backend-specific would be vllm. Any idea about it?

gcanlin · 2026-05-24T16:10:53Z

@aoshen02 I push more cleaning. Please take another look when you have time. Thx!

aoshen02 · 2026-05-25T01:49:30Z

    runs-on: self-hosted
    container:
-      image: lmsysorg/sglang:v0.5.0rc0-cu126
+      image: slimerl/slime:latest


We can use new image for now.

aoshen02 · 2026-05-25T01:50:13Z

Thanks for the pr. So I have a general question that will we rename sglang to rollout instead of vllm directly, I've seen some are vllm while some are rollout.

I was thinking about it. Currently, this PR is making some general names as rollout and some backend-specific would be vllm. Any idea about it?

I think we should just use vllm by default to maintain simplicity and consistency. As we previously discussed, the philosophy of this framework is to keep minimal encapsulation and stick to a single rollout and training backend."

aoshen02 · 2026-05-25T01:56:22Z

-        "--sglang-mem-fraction-static 0.6 "
-        f"--sglang-cuda-graph-bs {' '.join(map(str, [1, 2, 4, 8] + list(range(16, 257, 8))))} "
-    )
+    rollout_engine_args = "--rollout-num-gpus-per-engine 1 " "--vllm-gpu-memory-utilization 0.6 "


should add cuda graph selection

aoshen02 · 2026-05-25T02:00:10Z

@@ -0,0 +1 @@
+


We can just remove this?

aoshen02 · 2026-05-25T02:01:06Z

        self.model_path = model_path or args.hf_checkpoint
        # Uniform Ray ``start_engines`` kwargs; unused when launching vLLM over HTTP.
-        self.sglang_overrides = sglang_overrides or {}
+        self.engine_overrides = engine_overrides or {}


We can just rename to vllm_overrides.

Yeah. Let me rename them all once.

aoshen02 · 2026-05-25T02:42:41Z

+        return out_w, out_s
+
+    def transform_scale_ue8m0(sf: torch.Tensor, mn: int, use_torch_impl: bool = False):
+        import deep_gemm.utils.layout


Please move it to the top

aoshen02 · 2026-05-25T02:46:39Z

@@ -0,0 +1 @@
+


We can just delete sglang_utils

aoshen02 · 2026-05-25T03:55:05Z

@@ -1,4 +1,24 @@
-"""Configuration dataclasses for SGLang engine deployment."""
+"""Deployment configuration dataclasses for the rollout engine.


I have a question here about why move to slime/utils/rollout_config.py instead of vllm_utils/vllm_config.py?

Have reverted it now. Before, I was trying to use the general name RolloutConfig so move it as well. But currently, we align with the name.

aoshen02 · 2026-05-25T04:02:30Z

@@ -50,7 +50,6 @@ NVCC_APPEND_FLAGS="--threads 4" \
 pip install git+https://github.com/fzyzcjy/torch_memory_saver.git@dc6876905830430b5054325fa4211ff302169c6b --no-cache-dir --force-reinstall


We should consider use official repo instead of their fork, fzyzcjy is their core contributor, we can replace these in other PR.

Agree. I will clean other dependencies in a follow-up PR.

- Replace "rollout engine"/"rollout backend"/"inference engine" hedges with "vLLM" in slime/ comments and CLI help strings where the goal is vLLM-specific (per PR #18 review §13). - Replace stale "pkill -9 sglang" prefixes in scripts/examples/tests with "pkill -9 -f \"vllm serve\"" (§11). sglang processes no longer exist after this PR; the vllm-serve pattern is what slime actually spawns. No behavior change: comments + help text + worker bootstrap pre-clean only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

PR #18 removed all sglang_router / sglang imports from slime/. The "--no-deps stub" install for sglang and sglang-router (and the version print in the fail-fast smoke) was kept defensively but is now dead weight. Remove the install line, drop ARG SGLANG_VERSION, and drop sglang from the import-smoke RUN line. Kept defensively: the "strip sglang from requirements.txt" filter, in case someone re-adds the line; the GB300-gated sgl-kernel pin, which is hardware-specific and unrelated to sglang the runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

PR #18 silently dropped 8 SGLANG_* engine env vars without vLLM equivalents. Most have no vLLM analogue or are vLLM defaults already (see pr18-review.md §14 mapping). The one correctness-relevant gap is SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_FALLBACK_VARIANT, whose vLLM equivalent is VLLM_BATCH_INVARIANT (vllm/envs.py:84,528 — consumed in 14 sites across attention backends (FA, FlexAttention, MLA, Triton), all_reduce / symm_mem comms, cascade-attention disable, and config). Before: --vllm-enable-deterministic-inference only added per-sample seed in vllm_rollout.GenerateState; engine kernels were still selected by batch shape, so the same prompt produced different logits depending on batch composition. After: same flag also exports VLLM_BATCH_INVARIANT=1 to the vllm serve subprocess, switching attention/comm/MM kernels to batch-invariant variants. Both seed and batch-invariant kernels are needed for true determinism. Help text on --vllm-enable-deterministic-inference updated to reflect the full contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

§5: examples/on_policy_distillation/run-qwen3-8B-opd.sh — change "--opd-type sglang" to "--opd-type vllm" (PR #18 already restricted arguments.py choices to {vllm, megatron}). §8: delete build_rollout_meta_trace_attrs + ROLLOUT_TRACE_META_KEYS from slime/utils/trace_utils.py. The function was renamed from build_sglang_meta_trace_attrs by PR #18, but its only caller (sglang_rollout.py:202) was deleted in the same PR. vllm_rollout.py spans do not reproduce this attribution; the SGLang meta_info schema (nested finish_reason.type, pd_prefill_/pd_decode_ PD fields) does not match vLLM's choices[0]/usage shape, so a clean port requires a non-trivial vllm-meta builder. Per PR review default (delete to keep diff small), drop both the function and its dedicated test; inline the equivalent dict in the trace-viewer test that incidentally used the function. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

PR #18 left two stale SGLang-isms in the consistent-hash routing path: 1. slime/rollout/vllm_rollout.py:423 checked args.router_policy == "consistent_hashing" (with -ing). vllm-router's policy enum is "consistent_hash" (no -ing) — see vllm_router_rs::router:: policy_from_str + RouterArgs.policy choices. The check never fired under a vLLM deployment. 2. The same branch set header X-SMG-Routing-Key (SGLang Model Gateway). vllm-router's ConsistentHashPolicy.extract_hash_key_from_headers recognizes x-session-id / x-user-id / x-tenant-id (see vllm_router_rs strings). X-SMG-Routing-Key was silently ignored. Update both: policy string to "consistent_hash", header to "x-session-id". Update the corresponding unit test (test_generate_consistent_hash_header) and the docstring on Sample.session_id in types.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…emm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…d infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…p TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts: # slime/backends/vllm_utils/arguments.py

…_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* tests + CI: complete sglang→vllm rename across tests/ and .github/ Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * on_policy_distillation: port from SGLang to vLLM /v1/completions Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * tests: drop duplicate smoke test updates from PR40 * test(update_weight_from_tensor): drop stale _apply_monkey_patch_torch_reductions patch The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):`` context in _run_update suppressed a helper call that PR #48 has since deleted from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime). After that PR lands the patched attribute won't exist and this line raises AttributeError. Remove it now so the test survives PR #48 merge. The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the fake sglang module is intentionally kept: on this branch the production code still imports it via ``from ..sglang import monkey_patch_torch_reductions`` (both update_weight_from_tensor._apply_monkey_patch_torch_reductions on PR #40's view of main, and hf_weight_iterator_direct.py at module level). Removing the stub here would break the test on PR #40 alone; it can be dropped in a follow-up once PR #48 finishes removing every import site. Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py`` all 6 pass with this change applied to gcl/pr18-tests-ci HEAD. * tests: drop duplicate top-level test_update_weight_from_tensor.py The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(vllm_config): use real get_model_url default endpoint /inference/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop on_policy_distillation.py from tests+CI PR (now owned by runtime PR #18) The OPD vLLM /v1/completions migration is a runtime change; it was folded into the core-runtime PR (#18). Restore this file to main here so the two PRs no longer overlap on it. #18 merges first, so this lands via #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop unit-test files now owned by runtime PR #18 test_vllm_config.py + the plugin_contracts tests are coupled to #18's runtime rename (they import vllm_config / vllm_rollout, which #18 creates). They live in #18; remove them here so the two PRs don't overlap. #18 merges first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: restore vLLM rollout args dropped during sglang→vllm rename The mechanical sglang→vllm rename dropped several rollout knobs instead of mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph capture caps, speculative decoding, expert parallel). Restore them using the mapping established by the converted production scripts on main (run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM AsyncEngineArgs: --sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N --sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c --sglang-ep-size N -> --vllm-enable-expert-parallel --sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}' Also: - glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs; --sglang-max-running-requests was 16) and split out cuda-graph capture. - fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*). - test_vllm_config: rename test_update_weights_default_true → test_update_weights_defaults_to_none (it asserts `is None`). Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head, moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy, disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer is driven by --prefill-num-servers + the --vllm-config prefill/decode topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(plugin_contracts): migrate from sglang_rollout to vllm_rollout The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(update_weight_from_tensor): drop stale slime…megatron_utils.sglang mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): use method=mtp (not eagle) in vllm speculative config The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ckpt): add --dist-ckpt-optim-fully-reshardable for PAO+offload save/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>

… test (#77) The hybrid router-arg convention (settled in #18/#40) keeps the vllm_ prefix only for the endpoint-locating flags --vllm-router-ip and --vllm-router-port; every other vllm-router knob is passed bare with a --router- prefix (dest router_*). The glm4.7 PD/mooncake integration test still passed the pre-hybrid name --vllm-router-request-timeout-secs, which is no longer a registered flag, so the 1200s timeout never reached router_args.request_timeout_secs. This test is GPU-only and outside the unit suite, so the rename sweep in #40 missed it. Rename the flag to --router-request-timeout-secs to match arguments.py (dest router_request_timeout_secs) and rollout.py's consumer. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

The built-in OPD reward_func (vime/rollout/on_policy_distillation.py) was rewritten in #18 to POST vime's disaggregated /inference/v1/generate contract (top-level token_ids + nested sampling_params, top-level prompt_logprobs in the response). But that route is mounted only on the disagg serve app, not on the stock vllm.entrypoints.openai.api_server that the OPD test (and example launch scripts) run as the teacher. The test points --rm-url at /v1/completions, and merged PR #62 documents the intended contract as /v1/completions with echo + prompt_logprobs=1 consuming choices[0].prompt_logprobs. The mismatch makes the teacher return HTTP 400, which resp.raise_for_status() turns into an aiohttp ClientResponseError whose CIMultiDictProxy headers fail to pickle across Ray, masking the real error as "can't pickle CIMultiDictProxy". - reward_func: send the OpenAI /v1/completions body (prompt=token_ids, max_tokens=1, temperature=0, echo=True, prompt_logprobs=1, skip_special_tokens=False); model only when --opd-teacher-model is set. - replace resp.raise_for_status() with an explicit non-200 -> RuntimeError carrying the status + body (picklable; surfaces the real teacher error). - post_process_rewards: read choices[0].prompt_logprobs (per-position shape is identical, so the lookup/length/trim logic is unchanged). Validated on gb200 (2 student + 1 teacher GPU), EXIT_RC=0: teacher HTTP 200, teacher_log_probs flow into opd_reverse_kl, full rollout+train+eval clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>

…ation) Bidirectional translation audit (OVER): the #18 port both injected "vLLM" into engine/rollout log strings AND paraphrased them. slime was engine-neutral here, so restore slime's EXACT wording (not just drop the engine word): vllm_engine.py: "Error resetting vLLM prefix cache: …" → "Error flushing cache: …" (slime verbatim) "Timeout while resetting vLLM prefix …"→ "Timeout while flushing cache." "Shutdown vLLM engine …" → "Shutdown engine …" "Simulating crash on vLLM engine …" → "Simulating crash on engine …" vllm_rollout.py: "vLLM rollout abort (pause) …" → "Abort request for %s" (slime verbatim) "Failed to pause/abort worker …" → "Failed to abort worker at %s: %s" (method name flush_cache and the /reset_prefix_cache endpoint are vLLM's real API and unchanged; the vime-only resume-after-drain logs have no slime counterpart and stay neutral. Legit vLLM strings — IPC/server_info/routing replay/MM/sleep-wake + "Use external SGLang engine"→vLLM — kept.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…VER-translation) Restore slime-verbatim neutral log wording where vime had over-translated by inserting "vLLM" into strings slime kept engine-agnostic: - "Shutdown vLLM engine" -> "Shutdown engine" - "Simulating crash on vLLM engine" -> "Simulating crash on engine" - "vLLM rollout: resuming workers..." -> "rollout: resuming workers..." - "Failed to resume vLLM worker" -> "Failed to resume worker" - "vLLM rollout abort (pause) for workers" -> "Abort request for %s" - "Failed to pause/abort worker" -> "Failed to abort worker at %s" flush_cache log strings dropped from this PR: #137 removes that retry loop entirely (vLLM /reset_prefix_cache always 200s), so there is nothing left to re-word there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…VER-translation) (#153) Restore slime-verbatim neutral log wording where vime had over-translated by inserting "vLLM" into strings slime kept engine-agnostic: - "Shutdown vLLM engine" -> "Shutdown engine" - "Simulating crash on vLLM engine" -> "Simulating crash on engine" - "vLLM rollout: resuming workers..." -> "rollout: resuming workers..." - "Failed to resume vLLM worker" -> "Failed to resume worker" - "vLLM rollout abort (pause) for workers" -> "Abort request for %s" - "Failed to pause/abort worker" -> "Failed to abort worker at %s" flush_cache log strings dropped from this PR: #137 removes that retry loop entirely (vLLM /reset_prefix_cache always 200s), so there is nothing left to re-word there. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ples (#39) * examples/scripts/slime_plugins: complete sglang→vllm rename + drop SGLang-only examples Split from PR #18. 68 files / +473 / -2180. What this PR does: (A) `examples/` (35→24 files after deletions) and `scripts/` (25 files): rename `--sglang-*` CLI flags to `--vllm-*` equivalents, rename `SGLANG_ARGS=(...)` → `VLLM_ARGS=(...)`, replace `sgl_router_ip` / `sgl_router_port` env var references with the `vllm_router_*` versions, update HF docker image refs, and similar mechanical renames across all training recipes (geo3k_vlm{,_multi_turn} including NPU variants, multi_agent, on_policy_distillation, retool, search-r1, train_infer_mismatch_helper) plus all top-level scripts/run-*.sh and scripts/low_precision/*.sh. (B) `slime_plugins/rollout_buffer/rollout_buffer_example.{py,sh}`: `args.sglang_router_{ip,port}` → `args.vllm_router_{ip,port}`, `pkill -9 sglang` → `pkill -9 vllm`. These are usage examples for the rollout-buffer plugin, not plugin core. (C) Delete SGLang-coupled examples (cannot run on vime, which is vLLM-only after #18 core): - `examples/strands_sglang/` (4 files): entire example targets the `strands-sglang` library (SGLang extension for the `strands` agentic scaffolding). No vLLM-equivalent shim exists upstream; re-introducing this example would require porting strands-sglang itself. - `examples/tau-bench/` (7 files): tau-bench's tool-call parser chain is `trainable_agents.py → openai_tool_adapter.py → sglang_tool_parser.py → sglang.srt.function_call.FunctionCallParser`. With sglang removed from the runtime, the import chain breaks at load time. PR #18 left `sglang_tool_parser.py` untouched and kept the `_sglang`-suffixed method names, so the example never actually worked on vime — the cleanest cut is full removal until someone wants to port to vLLM's tool-calling API as a separate PR. - `examples/README.md`: drop the two list entries that referenced `strands_sglang` and `tau-bench`. (D) `scripts/run-deepseek-r1.sh`: drop the `/sgl-workspace/nvshmem/install/lib/` path from `LD_LIBRARY_PATH`. This was a path that existed only inside the old SGLang docker image (`slimerl/sglang:dev`); vime ships from `vllm/vllm-openai`, where that directory doesn't exist — the `:`-suffixed path was silently ignored at best and risked masking real nvshmem locations at worst. Sgl residue after this PR (3 remaining): - `examples/search-r1/local_dense_retriever/{retrieval_server,download}.py` "Copyright 2023-2024 SGLang Team" headers — legitimate attribution for vendored code, kept. - `examples/retool/README.md` references HuggingFace model id `font-info/qwen3-4b-sft-SGLang-RL` — third-party identifier we cannot change. This PR is independent of the other splits (#38 docs, upcoming tests+CI, PR #18 core). Can land at any time. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * examples/scripts: restore flags dropped by mechanical sglang→vllm rename Reviewing PR #39 against the §6/§7 parameter-mirror table in pr18-post-fix-audit.md, three scripts still had flags that were either dropped silently during the mechanical sed pass, or had comment blocks mangled by the same pass. Mirror table (carry over from the audit): --sglang-cuda-graph-bs LIST → --vllm-cudagraph-capture-sizes LIST --sglang-ep-size N → --vllm-enable-expert-parallel (size auto-derived) --sglang-speculative-{algorithm,num-steps,eagle-topk,num-draft-tokens} → --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":N}' Restored flags: (1) examples/geo3k_vlm/run_geo3k_qwen35.sh: + --vllm-enable-expert-parallel + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 256) + --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":3}' The MTP speculative-decoding block was orphan-commented but had no replacement flag (the 4 spec-decode sglang flags were dropped without the unified config). (2) examples/multi_agent/run-qwen3-30B-A3B-multi-agent.sh: + --vllm-cudagraph-capture-sizes 1 2 4 8 $(seq 16 8 256) The original SGLANG_ARGS had cuda-graph-bs; the new VLLM_ARGS only kept gpu-memory-utilization, dropping cudagraph capture sizes. (3) scripts/low_precision/run-kimi-k2-Thinking-int4.sh: Comment block restoration. The mechanical pass produced corrupted nested comments (`# # --vllm-data-parallel-size 8 / # # # / #`) that lost the original "uncomment to enable DP-attention" intent. Replaced with clear "uncomment to enable" prompts for the two sglang-only-but-now-vllm-flag features (DP-attention via --vllm-data-parallel-size and DeepEP MoE via --vllm-all2all-backend). Verification: re-ran the rename-aware diff scan of every script — all SGLANG_ARGS items from main now have a corresponding VLLM_ARGS item (or are documented sglang-only and intentionally dropped). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * examples/tau-bench: port to vLLM-native tool calling, restore example Reverts the deletion of examples/tau-bench/ from PR #39 commit ad03295 and re-implements the SGLang-coupled tool-call parsing using vLLM's Hermes/Qwen-2.5 tool-call format (as per user direction: "用 vLLM 自带的 tool-calling就行，看vllm源码"). Changes vs the pre-#18 baseline: 1. `examples/tau-bench/sglang_tool_parser.py` → DELETED. It imported `from sglang.srt.function_call.function_call_parser import FunctionCallParser` and `from sglang.srt.managers.io_struct import Function, Tool`. With sglang removed from the vime runtime, this import broke `examples.tau-bench` at load time. 2. `examples/tau-bench/tool_parser.py` → NEW (self-contained, no external sglang/vllm import at runtime). Mirrors vLLM's `Hermes2ProToolParser.extract_tool_calls` (`reference/vllm/vllm/tool_parsers/hermes_tool_parser.py`) for the non-streaming case. Same regex `<tool_call>(.*?)</tool_call>|<tool_call>(.*)` (DOTALL), same JSON-payload shape (`{"name": ..., "arguments": ...}`), same Qwen-2.5 chat template output. Public surface preserves `parse_tools(response, tools, parser)` returning `{"normal_text": str, "calls": [{"name", "parameters"}]}`, so downstream `openai_tool_adapter.py` is untouched in shape. Smoke-tested locally: no tool call, single tool call, malformed JSON (swallowed), multiple tool calls, invalid parser id. 3. `examples/tau-bench/openai_tool_adapter.py`: - Import: `from sglang_tool_parser` → `from tool_parser`. - Rename `_call_to_action_sglang` → `_call_to_action`. - Updated docstrings/comments to drop sglang references. 4. `examples/tau-bench/trainable_agents.py`: - `from slime.rollout.sglang_rollout import GenerateState` → `from slime.rollout.vllm_rollout import GenerateState` (vime moved `GenerateState` to the vllm rollout module). - URL: `http://{sglang_router_ip}:{port}/generate` → `http://{vllm_router_ip}:{port}/v1/completions`. - Payload: `{"text", "sampling_params": {...}}` → `{"prompt", **sampling_params}` (OpenAI completions shape). - Response parsing: `output["text"]` / `output["meta_info"][ "finish_reason"]["type"] == "abort"` → `choice = output[ "choices"][0]; choice["text"]` with abort detection on any `finish_reason` outside `{None, "stop", "length", "tool_calls"}`. - `rollout_args` defaults: `sglang_router_*` → `vllm_router_*`. - `sampling_params` default `max_new_tokens` → `max_tokens` (OpenAI naming). - Rename free function `call_to_action_sglang` → `call_to_action`. - Docstring/comment cleanups dropping sglang references (kept the verl `sglang_multiturn` URL — that's a path on verl's docs site, a third-party identifier). 5. `examples/tau-bench/run_qwen3_4B.sh`: - `pkill -9 sglang` → `pkill -9 -f "vllm serve"`. - `SGLANG_ARGS=(--sglang-mem-fraction-static 0.7 ...)` → `VLLM_ARGS=(--vllm-gpu-memory-utilization 0.7 ...)`. 6. `examples/README.md`: restore the tau-bench list entry that ad03295 removed. Verification: - `grep -E "sglang|sgl_" examples/tau-bench/` returns no real hits (only a verl docs URL containing "sglang_multiturn" as a path, legitimate third-party reference). - `python3 -c "import ast; ast.parse(open('examples/tau-bench/ trainable_agents.py').read())"` → OK. - Local smoke test on `tool_parser.py` covers 5 cases (no call, single call, malformed, multi-call, bad parser id). End-to-end runtime correctness on a test machine is the caller's responsibility — the runtime payload/response shape is a best-effort port; if `vllm-router` surfaces an abort condition differently from what this code assumes, that branch may need adjustment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * examples/tau-bench: rename tool_parser.py → vllm_tool_parser.py Adds a 'vllm' prefix so the filename signals where the parser was ported from (vLLM's Hermes2ProToolParser) and avoids the ambiguous generic name. Updated callers: - openai_tool_adapter.py: 'from tool_parser' → 'from vllm_tool_parser' - trainable_agents.py docstring: 'tool_parser.parse_tools' → 'vllm_tool_parser.parse_tools' Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * examples/tau-bench: drop entire example After porting tau-bench to vLLM-native tool calling (91ba909 + 85ba197), the user opted to remove the example outright rather than maintain a tau-bench port. The justification: - vime is not actively running tau-bench in CI; the example's true test surface is local-only on the contributor's machine. - openai_tool_adapter.py was ~85% dead code with only a 1-line shim around vllm_tool_parser.parse_tools actually exercised by callers — once dead code is removed, the example is too thin to justify carrying. - Anyone who wants tau-bench can recover it from git history (last present at commit 91ba909) and port further from there. Deleted (7 files): - examples/tau-bench/README.md - examples/tau-bench/generate_with_tau.py - examples/tau-bench/openai_tool_adapter.py - examples/tau-bench/run_qwen3_4B.sh - examples/tau-bench/tau1_mock.py - examples/tau-bench/trainable_agents.py - examples/tau-bench/vllm_tool_parser.py Also dropped the tau-bench list entry from examples/README.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…version-with-data) (#48) * refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) Background ---------- After PR #22 introduced the colocated CUDA IPC path, vime ended up with three ``update_weights*`` RPC entry points whose ``_weight_version`` bookkeeping was inconsistent: - ``update_weights_from_distributed`` (NCCL path): writes ``_weight_version`` inside the RPC, version travels with data — slime-style. - ``update_weights`` (IPC path, called from ``IPCWeightTransferEngine.trainer_send_weights``): forwarded vLLM's ``IPCWeightTransferUpdateInfo`` to ``/update_weights`` over HTTP but never recorded ``_weight_version`` — vLLM's payload schema does not carry it. - ``update_weights_from_tensor`` (PR #18 legacy entry): kept a SGLang-ish ``serialized_named_tensors`` payload and wrote ``_weight_version``, but had no callers in main. The IPC gap was the root cause of #41-era's "Weight version mismatch! Engine: /root/models/<...>, Updater: N" failure on every colocated test with ``--ci-test`` (fixed in #45 by piggybacking ``weight_version`` onto ``finish_weight_update``). slime's design avoids this entirely: both IPC and distributed call ``engine.update_weights_from_tensor.remote(..., weight_version=N)`` — same RPC name across both repos, with version travelling alongside the data in the same RPC. This PR ------- Rewire vime's IPC path to match slime's interface: 1. ``vllm_engine.update_weights_from_tensor`` is now the IPC entry point. Signature ``(update_info: dict, weight_version: str | None, flush_cache)``; payload carries vLLM's ``IPCWeightTransferUpdateInfo`` (names / dtype_names / shapes / ipc_handles), the trainer constructs it with ``reduce_tensor`` from ``torch.multiprocessing.reductions``. Records ``_weight_version`` only after the POST succeeds — mirrors ``update_weights_from_distributed``'s post-POST ordering so a failed transfer never advances the engine's tracked version. 2. Delete ``vllm_engine.update_weights`` — was the vLLM ``IPCWeightTransferEngine.trainer_send_weights`` entry, no longer used after step 4 below. 3. Delete ``vllm_engine._run_vllm_weight_update`` — dead helper that only ``update_weights_from_tensor``'s old SGLang-ish path called. 4. Revert ``finish_weight_update`` to a stateless POST — ``_weight_version`` now lives in the data-carrying RPC, so the bookend no longer needs to piggyback a kwarg. (Undoes the kwarg added in #45.) 5. Replace ``IPCWeightTransferEngine.trainer_send_weights(...)`` calls in ``_send_hf_chunk_via_ipc`` with direct ``engine.update_weights_from_tensor.remote(update_info=..., weight_version=...)`` for both slot_size paths (slot_size==1 and slot_size>1). vime keeps reusing vLLM's ``reduce_tensor`` for IPC handle creation (via ``_build_ipc_update_info_from_named_tensors``) — only the dispatch is ours — so we don't fork the vLLM IPC protocol, just route through our own RPC surface. Why not just keep #45's piggyback? - #45 worked but coupled version bookkeeping to the lifecycle hook ``finish_weight_update`` instead of the data RPC. The wire shape doesn't match slime's, and a new IPC-style entry point added later would have to remember to also write ``_weight_version`` — exactly the trap PR #22 fell into. Centralising the write inside the data RPC removes the trap. Unit tests ---------- - ``RecordingVLLMEngine`` learns ``update_weights_from_tensor`` so engine RPC call recording stays complete. - Renamed ``test_trainer_send_weights_uses_single_llm_handle_per_rank`` -> ``test_send_via_ipc_dispatches_update_weights_from_tensor_with_version``, asserts the new RPC name + kwargs (``update_info``, ``weight_version``) and that ``finish_weight_update`` is now stateless (no kwargs). - Added ``test_update_weights_from_tensor_posts_ipc_update_info_and_records_version``: asserts ipc_handles get cloudpickle'd into ipc_handles_pickled, metadata fields pass through, ``_weight_version`` advances on POST success. - Added ``test_update_weights_from_tensor_does_not_advance_version_on_failure``: asserts POST failure does not advance ``_weight_version`` (matches the same post-POST ordering review note from #45). Pre-existing test failures in tests/unit/backends/vllm_utils/test_vllm_engine.py (``_weight_transfer_http_timeout``, ``_response_json_or_fallback``, ``server_host``) are unchanged from main — main has 8 failed / 24 passed, this PR has 8 failed / 26 passed (the two added tests). Not in scope here. Signed-off-by: aoshen02 <aoshen@inferact.ai> * fix(weight-sync): correct IPC slot leader gating and gather group Two regressions in the previous commit only fire when Megatron TP != rollout-num-gpus-per-engine (e.g. parallel-check sweeps Megatron TP=1 with rollout TP=2). Both surface via ``tests/test_qwen3_0.6B_parallel_check.py``. Bug 1: leader gating wrong reference group ------------------------------------------ ``connect_rollout_engines`` used:: if mpu.get_tensor_model_parallel_rank() == 0: self._ipc_engine_coordinator = True The intent was "TP rank 0 within the engine GPU slot", but ``mpu.get_tensor_model_parallel_rank()`` is the Megatron TP rank, not the engine-slot rank. When Megatron TP=1, every trainer rank sees ``tp_rank=0`` and becomes a coordinator. For slot_size > 1, both ranks in the slot then call ``start_weight_update`` → the second call explodes:: Worker failed with error 'start_weight_update called while a weight update is already active. Call finish_weight_update first.' Fix: gate on ``rank == start`` (lowest trainer rank in the engine GPU range). Unique per slot regardless of Megatron parallelism. Bug 2: gather group wrong scope ------------------------------- ``_send_hf_chunk_via_ipc`` used ``mpu.get_tensor_model_parallel_group()`` to all_gather IPC payloads from peers in the engine slot. Again Megatron TP group ≠ engine slot when their world sizes differ. The merge then only had the coordinator's own UUID; downstream workers reading a different physical GPU got:: ValueError: IPC handle not found for GPU UUID <peer>. Available UUIDs: ['<coordinator>'] Fix: build per-slot process groups in ``connect_rollout_engines`` collectively (every trainer rank calls ``dist.new_group(slot_ranks)`` for every engine slot, keeps the one it belongs to). Use that group instead of Megatron's TP group for the gather and the trailing barrier. Validation ---------- Ran ``tests/test_qwen3_0.6B_parallel_check.py`` on 8×H200 with ``--num-rollout 2``. The test sweeps tp_size ∈ {1, 2, 4, 8} × pp_size ∈ {1, 2, 4} × cp_size ∈ {1, 2, 4, 8} for num_gpus ∈ {8, 4, 2} — every leg also uses ``--rollout-num-gpus-per-engine 2``, so the Megatron-TP=1 cases now exercise the new slot-group path. Pre-fix: fails on the very first Megatron-TP=1 config with bug 1 above; after bug 1 is patched, the next iteration fails with bug 2. Post-fix: the entire ~2-hour sweep completes with "Job succeeded" on every leg. Other tests already exercising IPC at Megatron-TP=2 == rollout-TP=2 are unaffected by this fix (``rank == start`` is equivalent to ``tp_rank == 0`` when the slot fits a single Megatron TP group). The following also pass post-fix as a sanity check: ``test_qwen3_4B_ppo``, ``test_qwen3_4B_ppo_train_critic_only``, ``test_qwen3_4B_ppo_disaggregate``, ``test_mimo_7B_mtp_only_grad``, ``test_moonlight_16B_A3B``, ``test_quick_start_glm4_9B``, ``test_qwen2.5_0.5B_{short,async_short,debug_rollout_then_train, ppo_critic_only_short}``, ``test_qwen3.5_0.8B_gsm8k_{short,async_short}``. Signed-off-by: aoshen02 <aoshen@inferact.ai> 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(weight-sync): use gloo backend for ipc payload gather, add multi-gpu test Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(weight-sync): drop dead _apply_monkey_patch_torch_reductions calls The two _apply_monkey_patch_torch_reductions() call sites in this file (trainer send + vLLM worker hijack) are no-ops on the IPC path here: 1. We route IPC handles by physical GPU UUID dict key (set on the trainer side at _build_ipc_update_info_from_named_tensors via _current_gpu_uuid() from torch.cuda.get_device_properties().uuid). The receiver looks up by its own UUID, independent of args[6]. 2. vLLM's IPCWeightTransferEngine.receive_weights unconditionally overwrites args[6] with the receiver's local device_index before calling rebuild_cuda_tensor. Whatever a torch reductions patch encodes into args[6] is therefore discarded. The patch was the historical mechanism (sglang upstream) for translating device indices across CUDA_VISIBLE_DEVICES boundaries by stuffing UUID strings into args[6]. Our UUID-keyed dict + vLLM's explicit device_index override accomplish the same thing without the global torch reductions mutation. Also expand the _build_ipc_update_info_from_named_tensors docstring to spell out the UUID-keyed routing contract so future readers don't have to chase this through git history. Side effect: hf_weight_iterator_direct.py also calls monkey_patch_torch_reductions() at module-collective time. That call site is similarly decorative (only NCCL broadcast / all_gather collectives run there, no cross-process pickling) but lives outside this file's scope and is not touched here. Tracked alongside #29. * refactor(weight-sync): finish removing monkey_patch_torch_reductions dead code Follow-up to 39bf899 (deleted _apply_monkey_patch_torch_reductions from update_weight_from_tensor.py). With that helper gone, two more references are now dead in the vime IPC weight-transfer path: 1. hf_weight_iterator_direct.py:48 called monkey_patch_torch_reductions() at the top of _get_megatron_full_params(). On vime this never has effect on the IPC handle path: _get_megatron_full_params only runs NCCL broadcast/all_gather collectives (no cross-process pickling), and the chunks it returns are subsequently sent via PR #48's UUID-keyed {gpu_uuid: reduce_tensor(weight)} dict that vLLM's receiver routes by physical UUID + explicit args[6] overwrite. The call survives in slime/ miles upstream because their downstream path pickles tensors through sglang's MultiprocessingSerializer.serialize (ForkingPickler → reduce_tensor), where the patched encoding/decoding does real work; PR #48 does not use that pipeline, so the call here was incidentally inherited rather than functionally required. 2. slime/backends/megatron_utils/sglang.py's monkey_patch_torch_reductions re-export + __all__ entry now have no remaining importers in vime. Remove them. Also (this commit, B): 3. vllm_engine.py's update_weights_from_tensor docstring referred to "closures injected by _apply_monkey_patch_torch_reductions" as the reason for cloudpickle. That helper is gone; the cloudpickle is still correct because reduce_tensor returns a (rebuild_fn, args) tuple where the rebuild_fn is a module-level callable that JSON can't serialise. Update the docstring to reflect the actual reason. Net behaviour: identical — the deletions remove dead code paths. The patch_torch shim itself is still importable for callers outside vime (none currently in this tree). Cross-references: - #29 — issue documenting the no-op stub - 39bf899 — prior commit that deleted the helper from update_weight_from_tensor.py * docs(weight-sync): drop sglang refs + add IPC call-stack notes - update_weight_from_tensor.py module docstring: rewrite from a pure vLLM perspective. Step (2) now spells out the merged {uuid_G0: handle, uuid_G1: handle, ...} dict + collective_rpc fan-out inside the vLLM server, and step (3) makes version-with-data atomicity explicit. Drop the "match slime's sglang_engine signature" framing. - vllm_engine.update_weights_from_tensor: collapse the long sglang-vs-vLLM compare block into a focused docstring describing what the POST does and why ipc_handles needs cloudpickle. Add a Chinese call-stack walkthrough (trainer → ★this method★ → server collective_rpc → per-TP worker receive_weights) and note that `node_rank != 0` is a dead branch since VLLMEngine pins node_rank=0 (see PR #48 review comment for the follow-up cleanup). - Hoist base64/cloudpickle imports to module scope so the hot path no longer pays the per-call import overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> --------- Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: SamitHuang <285365963@qq.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: SamitHuang <285365963@qq.com> Co-authored-by: Cursor <cursoragent@cursor.com>

* docs: complete sglang→vllm rename across docs/ + README Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area (docs / examples / tests+CI / core runtime). See pr18-split-plan.md. 53 files. Pure documentation: no runtime code change. What this PR does: - docs/en/ + docs/zh/ (49 files): rename sglang-config.md → vllm-config.md (1 each side); update all --sglang-* CLI flag examples to their --vllm-* equivalents; rename references to SGLang as the rollout backend → vLLM; fix three broken upstream URLs (each replacement verified to load): * blog.vllm.ai/2025/10/26/distributed-batch-invariance.html → vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference * docs.vllm.ai/.../features/spec_decode.html (301 → 404) → docs.vllm.ai/.../features/speculative_decoding/ * github.com/vllm-project/vllm-router (404; repo does not exist) → github.com/vllm-project/router (the actual router repo) - docs/_static/image/ (3 files): rename sglang_config.png → vllm_config.png; refresh arch.png to the vllm-router architecture. - docs/en/advanced/reproducibility.md and docs/zh/advanced/reproducibility.md: the en and zh files had their content swapped (en had Chinese body, zh had English body). Restored the correct mapping. - docs/{en,zh}/blogs/introducing_slime.md: remove the "Special thanks to the AMD GenAI - Foundation Model Team for Day-1 AMD hardware support" line. With AMD support removed in #36, the acknowledgement implied ongoing support that no longer exists. - README.md / README_zh.md: expanded vLLM router argument prefix explanation (--router-* for vllm-router native flags vs --vllm-router-* for vime-side orchestration knobs); remove stale link to docs/zh/advanced/ slime_vllm_backend_design_v1.md which was deleted in commit fe700db. This PR is independent of the other splits and can land at any time without coordination. See pr18-split-plan.md §"合并顺序无强约束". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: scrub remaining sglang references and fix OPD teacher URL Follow-up to PR #38 review: 1. on-policy-distillation.md (en+zh): `--rm-url` example was `http://<TEACHER_IP>:<TEACHER_PORT>/v1/completions`. Updated to `/inference/v1/generate` — vime's vLLM rollout uses this endpoint by default; the teacher should match. 2. Removed SGLang-comparison advisory blocks. Treating SGLang as if it never existed in vime: - docs/{en,zh}/examples/deepseek-r1.md: - Drop ⚠️ "originally written for SGLang backend, check VLLM_ARGS against arguments.py" admonition - Drop "The original SGLang version of this example used large-EP inference (EP64, DP attention, DeepEP); confirm flag availability" - docs/{en,zh}/examples/{glm4.7-30B-A3B,glm4.7-355B-A32B,qwen3-30B-A3B}.md: Drop references to "SGLang's --enable-dp-attention" / "sglang `--enable-dp-attention`". The doc now just describes the vLLM configuration (DP on attention + EP on experts) without invoking SGLang as a reference point. - docs/{en,zh}/blogs/introducing_slime.md: - Drop parenthetical "(originally as an SGLang-native framework; this fork has been adapted to vLLM)" in the article subtitle - Rewrite the "RL-specific workloads" paragraph from past-tense "the original SGLang version upstreamed patches; vLLM port replays..." to present-tense "slime builds on vLLM's /inference/v1/generate ... to support MoE weight updates etc." 3. Final grep result: `grep -E 'sgl|SGL' docs/` returns matches only in docs/{en,zh}/blogs/release_v0.1.0.md (the historical slime v0.1.0 release blog with 6 SGLang mentions each, kept pending separate decision on whether to remove the blog entirely). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: remove historical slime blog posts + clean up index.rst toctree Per user direction "当作sglang完全不存在" — remove docs/{en,zh}/blogs/ in its entirety, plus orphan toctree entries: Removed files (6): - docs/en/blogs/release_v0.1.0.md (slime upstream v0.1.0 release notes; sourced from zhihu; tightly coupled to SGLang-era engineering, 6 sgl mentions) - docs/zh/blogs/release_v0.1.0.md (same, Chinese) - docs/en/blogs/introducing_slime.md (vision blog from lmsys.org 2025-07; SGLang-era origin doc, fully washed earlier but content overlaps with README, kept thin value-add) - docs/zh/blogs/introducing_slime.md (same, Chinese) - docs/_static/image/blogs/release_v0.1.0/cuda_vmm.png - docs/_static/image/blogs/release_v0.1.0/overrall.png Toctree cleanup (2 files): - docs/en/index.rst: * Drop "Hardware Platforms" toctree section — its only entry was `platform_support/amd_tutorial.md`, which #36 deleted from the tree but did not remove from the toctree (stale reference left behind on main). * Drop "Blogs" toctree section (now empty). - docs/zh/index.rst: * Drop "博客" toctree section (now empty). (zh side never had the Hardware Platforms section.) Verification: $ git grep -E 'sgl|SGL' -- docs/ # returns nothing — docs/ is now # 100% free of SGLang references. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: dedup arch.png — drop docs/_static/image copy, point to imgs/arch.png PR #28 (commit 0d249f9) updated `imgs/arch.png` to the vime architecture diagram. Vime had a second copy at `docs/_static/image/arch.png` referenced by Sphinx docs, which is inherently duplication of the same diagram at two paths. This PR's original diff modified the docs/_static/image/ copy (carried over from PR #18 upstream), but that perpetuates the duplication. Fix: delete the docs/_static/image/arch.png copy and have the two remaining references (docs/{en,zh}/advanced/vllm-config.md) point at the single canonical `imgs/arch.png` via relative path `../../../imgs/`. Single source of truth: 1 arch.png in the tree, used by README, README_zh, and both vllm-config.md docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: fix vllm_config.png wiring (redraw to match original sglang_config.png topology) The previous regenerate (8b3d1e3) had two wiring bugs vs the original upstream sglang_config.png: 1. MISSING: data buffer → megatron (data feeds training) 2. EXTRA: custom rollout generation → megatron (rollout shouldn't feed training data; that's the data buffer's job) It also had a visual bug — the 'prefill server' / 'decode server' / 'regular server' text labels were centred on the box and overlapping the GPU sprites inside. This commit re-runs matplotlib to produce the topologically correct graph: data buffer → megatron on the left column, custom rollout generation branching to the two server groups, and megatron → prefill server as the weight-sync edge into server group 1. Inner-server labels are now placed above the GPUs. Topology after this commit: data buffer ↔ custom rollout generation │ │ │ ↓ ↓ ↓ megatron ──────→ [server group 1] [server group 2] (vllm router) (vllm router) prefill+decode regular Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: mention TorchSpec for draft training * update architecture image. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: kaiyuan <kyxiezju@163.com>

* [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* tests + CI: complete sglang→vllm rename across tests/ and .github/ Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * on_policy_distillation: port from SGLang to vLLM /v1/completions Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * tests: drop duplicate smoke test updates from PR40 * test(update_weight_from_tensor): drop stale _apply_monkey_patch_torch_reductions patch The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):`` context in _run_update suppressed a helper call that PR #48 has since deleted from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime). After that PR lands the patched attribute won't exist and this line raises AttributeError. Remove it now so the test survives PR #48 merge. The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the fake sglang module is intentionally kept: on this branch the production code still imports it via ``from ..sglang import monkey_patch_torch_reductions`` (both update_weight_from_tensor._apply_monkey_patch_torch_reductions on PR #40's view of main, and hf_weight_iterator_direct.py at module level). Removing the stub here would break the test on PR #40 alone; it can be dropped in a follow-up once PR #48 finishes removing every import site. Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py`` all 6 pass with this change applied to gcl/pr18-tests-ci HEAD. * tests: drop duplicate top-level test_update_weight_from_tensor.py The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(vllm_config): use real get_model_url default endpoint /inference/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop on_policy_distillation.py from tests+CI PR (now owned by runtime PR #18) The OPD vLLM /v1/completions migration is a runtime change; it was folded into the core-runtime PR (#18). Restore this file to main here so the two PRs no longer overlap on it. #18 merges first, so this lands via #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop unit-test files now owned by runtime PR #18 test_vllm_config.py + the plugin_contracts tests are coupled to #18's runtime rename (they import vllm_config / vllm_rollout, which #18 creates). They live in #18; remove them here so the two PRs don't overlap. #18 merges first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: restore vLLM rollout args dropped during sglang→vllm rename The mechanical sglang→vllm rename dropped several rollout knobs instead of mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph capture caps, speculative decoding, expert parallel). Restore them using the mapping established by the converted production scripts on main (run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM AsyncEngineArgs: --sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N --sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c --sglang-ep-size N -> --vllm-enable-expert-parallel --sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}' Also: - glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs; --sglang-max-running-requests was 16) and split out cuda-graph capture. - fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*). - test_vllm_config: rename test_update_weights_default_true → test_update_weights_defaults_to_none (it asserts `is None`). Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head, moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy, disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer is driven by --prefill-num-servers + the --vllm-config prefill/decode topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(plugin_contracts): migrate from sglang_rollout to vllm_rollout The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(update_weight_from_tensor): drop stale slime…megatron_utils.sglang mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): use method=mtp (not eagle) in vllm speculative config The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ckpt): add --dist-ckpt-optim-fully-reshardable for PAO+offload save/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>

… test (#77) The hybrid router-arg convention (settled in #18/#40) keeps the vllm_ prefix only for the endpoint-locating flags --vllm-router-ip and --vllm-router-port; every other vllm-router knob is passed bare with a --router- prefix (dest router_*). The glm4.7 PD/mooncake integration test still passed the pre-hybrid name --vllm-router-request-timeout-secs, which is no longer a registered flag, so the 1200s timeout never reached router_args.request_timeout_secs. This test is GPU-only and outside the unit suite, so the rename sweep in #40 missed it. Rename the flag to --router-request-timeout-secs to match arguments.py (dest router_request_timeout_secs) and rollout.py's consumer. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

gemini-code-assist Bot reviewed May 20, 2026

View reviewed changes

knlnguyen1802 mentioned this pull request May 22, 2026

[feat] Support Colocated Weight Sync via CUDA IPC for vime #22

Merged

CalvinXKY reviewed May 22, 2026

View reviewed changes

gcanlin commented May 22, 2026

View reviewed changes

gemini-code-assist Bot reviewed May 22, 2026

View reviewed changes

aoshen02 reviewed May 22, 2026

View reviewed changes

aoshen02 mentioned this pull request May 23, 2026

ci: gate e2e jobs on pre-commit success #26

Merged

aoshen02 reviewed May 25, 2026

View reviewed changes

Comment thread slime/backends/sglang_utils/__init__.py Outdated

@@ -0,0 +1 @@

aoshen02 May 25, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can just remove this?

aoshen02 reviewed May 25, 2026

View reviewed changes

aoshen02 and others added 6 commits May 29, 2026 15:11

aoshen02 force-pushed the gcl/clean-sglang branch from 2d1a98b to 758112a Compare May 29, 2026 15:13

aoshen02 and others added 5 commits May 30, 2026 02:10

Merge remote-tracking branch 'origin/main' into pr18-merge

5a7ac10

# Conflicts: # slime/backends/vllm_utils/arguments.py

CalvinXKY merged commit 6e80b64 into main May 30, 2026
9 of 12 checks passed

This was referenced May 30, 2026

fix(tests): use bare --router-request-timeout-secs in glm4.7 mooncake test #77

Merged

[RFC] Qwen3-30B-A3B 8×H200 200-step convergence benchmark #79

Open

aoshen02 mentioned this pull request Jun 3, 2026

fix(opd): score teacher over /inference/v1/generate (text + multimodal) #141

Merged

aoshen02 mentioned this pull request Jun 5, 2026

RFC: Sync upstream slime PRs into vime (tracking) #107

Closed

aoshen02 deleted the gcl/clean-sglang branch June 8, 2026 14:17

	def transform_scale_ue8m0(sf: torch.Tensor, mn: int, use_torch_impl: bool = False):
	def transform_scale_ue8m0(sf: torch.Tensor, mn: int):

		def _rebuild_cuda_tensor_modified(*args):
		return reductions._rebuild_cuda_tensor_original(*args)

-    def _rebuild_cuda_tensor_modified(*args):
-        return reductions._rebuild_cuda_tensor_original(*args)
+    def _rebuild_cuda_tensor_modified(device, *args):
+        # Force the tensor to the current process's device to fix IPC mapping
+        return reductions._rebuild_cuda_tensor_original(torch.cuda.current_device(), *args)

	assert weight_block_size == [128, 128]
	assert list(weight_block_size) == [128, 128]

		@@ -1,4 +1,24 @@
		"""Configuration dataclasses for SGLang engine deployment."""
		"""Deployment configuration dataclasses for the rollout engine.

		@@ -50,7 +50,6 @@ NVCC_APPEND_FLAGS="--threads 4" \
		pip install git+https://github.com/fzyzcjy/torch_memory_saver.git@dc6876905830430b5054325fa4211ff302169c6b --no-cache-dir --force-reinstall

Uh oh!

Conversation

gcanlin commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 20, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 20, 2026

Choose a reason for hiding this comment

Uh oh!

gcanlin commented May 22, 2026

Uh oh!

CalvinXKY commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aoshen02 commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 22, 2026

Choose a reason for hiding this comment

Uh oh!

CalvinXKY commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aoshen02 commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aoshen02 commented May 22, 2026

Remaining sglang references after this PR

slime/ core

tests/

train.py / train_async.py

tools/

slime_plugins/

Two questions before proceeding

Uh oh!

gcanlin commented May 24, 2026

Uh oh!

gcanlin commented May 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aoshen02 commented May 25, 2026

gcanlin commented May 20, 2026 •

edited

Loading

Remaining `sglang` references after this PR

`slime/` core

`tests/`

`train.py` / `train_async.py`

`tools/`

`slime_plugins/`