tests + CI: complete sglang→vllm rename across tests/ and .github/ - #40
Merged
Conversation
Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request performs a comprehensive migration from SGLang to vLLM across the codebase, updating configuration schemas, command-line arguments, test scripts, and unit tests. It also introduces new unit tests for the UpdateWeightFromTensor class. The review feedback highlights critical test isolation issues in the newly added test files where global stubs are installed at import time, and identifies a mismatch between a test's name/docstring and its actual assertion in test_vllm_config.py.
|
|
||
|
|
||
| # Install stubs once at collection time so that the importlib fixture works. | ||
| _HF_ITER_STUB, _HF_BASE_CLS, _UPW_DIST_MOD, _IPC_MOD = _install_stubs() |
Contributor
There was a problem hiding this comment.
Installing stubs and modifying sys.modules and torch.distributed globally at the module level (during import/collection time) violates test isolation. Since pytest imports all test files during the collection phase, this code will execute even when running unrelated tests, permanently stubbing critical modules and potentially causing flaky test failures or silent bugs in other parts of the test suite.\n\nConsider refactoring this to use a module-scoped pytest fixture that handles the mocking and automatically restores the original state of sys.modules and torch.distributed after the tests in this module complete.
| return hf_iter_stub, upw_dist_mod | ||
|
|
||
|
|
||
| _HF_ITER_STUB, _UPW_DIST_MOD = _install_stubs() |
Contributor
There was a problem hiding this comment.
Similar to the other test file, installing stubs and modifying sys.modules and torch.distributed globally at the module level during import/collection time violates test isolation. This can cause side effects and flaky failures in other tests running in the same pytest session.\n\nPlease refactor this to use a module-scoped pytest fixture to safely mock these modules and restore the original environment after the tests finish.
| config = VllmConfig.from_yaml(path) | ||
| assert len(config.models) == 1 | ||
| assert config.models[0].update_weights is True | ||
| assert config.models[0].update_weights is None |
Contributor
There was a problem hiding this comment.
There is a mismatch between the test name/docstring and the actual assertion. The test is named test_update_weights_default_true and the docstring states that models without explicit update_weights should default to True, but the assertion checks that it is None.\n\nPlease update the test name and docstring to match the new behavior where the default is None.
Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
aoshen02
force-pushed
the
gcl/pr18-tests-ci
branch
from
May 27, 2026 03:44
bbbb79f to
7d0ecd7
Compare
# Conflicts: # tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py
Collaborator
Author
|
/gemini review |
Contributor
There was a problem hiding this comment.
Code Review
This pull request migrates the rollout and distillation engine from SGLang to vLLM across the codebase, updating configurations, command-line arguments, and tests. In particular, On-Policy Distillation (OPD) is updated to use vLLM's /v1/completions endpoint. Feedback focuses on improving robustness when parsing API responses (such as defensively checking dictionary structures and list indices), ensuring resource cleanup when pinging health endpoints, and addressing mismatches between updated tests and the underlying implementation (specifically regarding router arguments and consistent hashing headers).
Comment on lines
+123
to
+124
| choice = reward["choices"][0] | ||
| plp = choice.get("prompt_logprobs") or [] |
Contributor
There was a problem hiding this comment.
Directly accessing reward["choices"][0] is risky because reward might be None, not a dictionary, or contain an empty "choices" list if the API call failed or returned an unexpected payload. We should defensively guard against these cases to prevent TypeError, KeyError, or IndexError.
Suggested change
| choice = reward["choices"][0] | |
| plp = choice.get("prompt_logprobs") or [] | |
| choices = reward.get("choices") if isinstance(reward, dict) else None | |
| choice = choices[0] if choices else {} | |
| plp = choice.get("prompt_logprobs") or [] |
Comment on lines
+259
to
+261
| assert "vllm_router_ip" in args_mod._VIME_ORCHESTRATION_DESTS | ||
| assert "vllm_router_port" in args_mod._VIME_ORCHESTRATION_DESTS | ||
| assert "vllm_router_request_timeout_secs" in args_mod._VIME_ORCHESTRATION_DESTS |
Contributor
There was a problem hiding this comment.
This assertion will fail on this branch because _VIME_ORCHESTRATION_DESTS in slime/backends/vllm_utils/arguments.py still contains the old un-prefixed names ("router_ip", "router_port", "router_request_timeout_secs") instead of the new prefixed ones. Additionally, because of this mismatch, these orchestration dests won't be filtered out during CLI argument forwarding, causing the vllm serve subprocess to fail to launch with unrecognized argument errors. Please ensure _VIME_ORCHESTRATION_DESTS is updated accordingly.
Comment on lines
+560
to
+567
| _rollout_args(router_policy="consistent_hash"), | ||
| sample, | ||
| _default_sampling_params(), | ||
| ) | ||
| ) | ||
|
|
||
| headers = post_mock.await_args_list[0].kwargs.get("headers") | ||
| assert headers == {"X-SMG-Routing-Key": "sess-42"} | ||
| assert headers == {"x-session-id": "sess-42"} |
Contributor
There was a problem hiding this comment.
This test will fail because slime/rollout/vllm_rollout.py still expects "consistent_hashing" and sets "X-SMG-Routing-Key" in the headers, whereas this test has been updated to use "consistent_hash" and expect "x-session-id". Please make sure the implementation in vllm_rollout.py is aligned with these changes.
Comment on lines
+90
to
+91
| if pos_entry is None: | ||
| return 0.0 |
Contributor
There was a problem hiding this comment.
Using isinstance(pos_entry, dict) is more robust than checking pos_entry is None, as it safely handles any unexpected non-dictionary types (like lists or strings) that might be returned by the server, preventing potential AttributeErrors.
Suggested change
| if pos_entry is None: | |
| return 0.0 | |
| if not isinstance(pos_entry, dict): | |
| return 0.0 |
Comment on lines
+70
to
73
| req = urllib.request.urlopen(f"http://{TEACHER_HOST}:{TEACHER_PORT}/health", timeout=2) | ||
| if req.status == 200: | ||
| print(f"Teacher sglang server is ready on GPU {teacher_gpu}") | ||
| print(f"Teacher vLLM server is ready on GPU {teacher_gpu}") | ||
| return process |
Contributor
There was a problem hiding this comment.
Use a with statement when calling urllib.request.urlopen to ensure the connection/socket is properly closed, preventing potential resource leaks during the retry loop.
Suggested change
| req = urllib.request.urlopen(f"http://{TEACHER_HOST}:{TEACHER_PORT}/health", timeout=2) | |
| if req.status == 200: | |
| print(f"Teacher sglang server is ready on GPU {teacher_gpu}") | |
| print(f"Teacher vLLM server is ready on GPU {teacher_gpu}") | |
| return process | |
| with urllib.request.urlopen(f"http://{TEACHER_HOST}:{TEACHER_PORT}/health", timeout=2) as req: | |
| if req.status == 200: | |
| print(f"Teacher vLLM server is ready on GPU {teacher_gpu}") | |
| return process |
CalvinXKY
reviewed
May 27, 2026
| vllm_pipeline_parallel_size=1, | ||
| rollout_num_gpus_per_engine=4, | ||
| router_ip=None, | ||
| vllm_router_ip=None, |
Collaborator
There was a problem hiding this comment.
If we want to move to --vllm-router-*, could we change arguments.py + all call sites in the same PR so docs/tests/runtime stay consistent?
Collaborator
Author
There was a problem hiding this comment.
I think in this case then we will need to merge these two into one pr, which is also ok after I review it.
…_reductions patch
The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):``
context in _run_update suppressed a helper call that PR #48 has since deleted
from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime).
After that PR lands the patched attribute won't exist and this line raises
AttributeError. Remove it now so the test survives PR #48 merge.
The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the
fake sglang module is intentionally kept: on this branch the production code
still imports it via ``from ..sglang import monkey_patch_torch_reductions``
(both update_weight_from_tensor._apply_monkey_patch_torch_reductions on
PR #40's view of main, and hf_weight_iterator_direct.py at module level).
Removing the stub here would break the test on PR #40 alone; it can be
dropped in a follow-up once PR #48 finishes removing every import site.
Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py``
all 6 pass with this change applied to gcl/pr18-tests-ci HEAD.
This was referenced May 28, 2026
Resolve conflict in .github/workflows/pr-test.yml by regenerating from pr-test.yml.j2 so the renamed e2e-test-vllm-config job keeps the 'needs: pre-commit' gate added on main (#26). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aoshen02
commented
May 29, 2026
|
|
||
| if sample.multimodal_inputs and sample.multimodal_inputs.get("images"): | ||
| image_data = sample.multimodal_inputs["images"] | ||
| # vLLM accepts multimodal inputs via the chat completions endpoint |
aoshen02
commented
May 29, 2026
| Note: The reward_func calls the teacher server which returns token-level log-probs. | ||
| For pure on-policy distillation without task rewards, we return 0.0 for each sample. | ||
| The actual learning signal comes from the OPD KL penalty applied in compute_advantages_and_returns. | ||
| 1. Extracts teacher log-probs from the teacher's `/v1/completions` |
The mechanical sglang→vllm rename dropped several rollout knobs instead of
mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph
capture caps, speculative decoding, expert parallel). Restore them using the
mapping established by the converted production scripts on main
(run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM
AsyncEngineArgs:
--sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N
--sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c
--sglang-ep-size N -> --vllm-enable-expert-parallel
--sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}'
Also:
- glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs;
--sglang-max-running-requests was 16) and split out cuda-graph capture.
- fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*).
- test_vllm_config: rename test_update_weights_default_true →
test_update_weights_defaults_to_none (it asserts `is None`).
Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head,
moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy,
disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer
is driven by --prefill-num-servers + the --vllm-config prefill/decode topology.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aoshen02
added a commit
that referenced
this pull request
May 29, 2026
The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuilt against current main so the PR contains only the SGLang runtime
removal -- the docs / tests-ci / examples / scripts / docker portions were
split into separate PRs that have since merged.
- Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py,
rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all
docker/**/sglang.patch files.
- Rename the rollout config module sglang_utils/sglang_config.py ->
vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config ->
_resolve_vllm_config, --sglang-config -> --vllm-config); inline the
GPU_MEMORY_TYPE_* constants in rollout.py.
- Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported
through the sglang shim; repoint quantizer_fp8 to it.
- Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead
sglang-router dependency from requirements.txt.
- Finish the SGLang->vLLM rename in the runtime so it is internally consistent and
matches the tests landing in the tests/CI PR:
* router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout);
* get_model_url reads vllm_model_routers (aligning with rollout.py);
* --opd-type sglang -> vllm; engine_overrides rename;
* sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference,
wired to a real --vllm-enable-deterministic-inference flag (exports
VLLM_BATCH_INVARIANT=1);
* consistent_hash session-id routing uses vllm-router's x-session-id header;
* drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings.
- Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the
plugin-contract tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ocess cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…emm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d infra
tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword
profile/replay helpers to vLLM and map analyzer hints to vLLM flags
(--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments
SGLang -> vLLM.
build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300
sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale.
docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real
multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/
ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404
(CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already
CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker
selects the arm64 image automatically on an ARM host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of
the inherited sgl-project.github.io repo. .claude/skills/*: update the dead
`slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real
default is slime.rollout.vllm_rollout.generate_rollout).
justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it
equalled the Dockerfile default; the multi-arch manifest already resolves
arm64). train{,_async}.py: drop stray "the" in the W&B comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p TP
launch_server_process / _init_normal derived tensor-parallel size and
CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring
the per-engine num_gpus_per_engine already carried on the VLLMEngine actor.
A ServerGroup configured with num_gpus_per_engine greater than the global
flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync
rendezvous sized world_size from engine_gpu_counts (the per-group value).
The two disagreed: the trainer waited for a rank the under-sized engine
never started, so init_weight_transfer_engine hung for 300s
("3/4 clients joined") and the job failed.
Honor the per-engine num_gpus_per_engine at launch, falling back to the
global flag when unset (matches the SGLang path and PR #66's
_compute_server_args).
Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now
launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s
(was a 301s timeout), and rollout+eval proceed.
AI assistance (Claude Code) was used for this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # slime/backends/vllm_utils/arguments.py
…ave/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # .github/workflows/conda-ci.yml # tests/unit/backends/vllm_utils/conftest.py # tests/unit/backends/vllm_utils/test_arguments.py
CalvinXKY
pushed a commit
that referenced
this pull request
May 30, 2026
* [Clean] Remove SGLang runtime code
Rebuilt against current main so the PR contains only the SGLang runtime
removal -- the docs / tests-ci / examples / scripts / docker portions were
split into separate PRs that have since merged.
- Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py,
rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all
docker/**/sglang.patch files.
- Rename the rollout config module sglang_utils/sglang_config.py ->
vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config ->
_resolve_vllm_config, --sglang-config -> --vllm-config); inline the
GPU_MEMORY_TYPE_* constants in rollout.py.
- Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported
through the sglang shim; repoint quantizer_fp8 to it.
- Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead
sglang-router dependency from requirements.txt.
- Finish the SGLang->vLLM rename in the runtime so it is internally consistent and
matches the tests landing in the tests/CI PR:
* router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout);
* get_model_url reads vllm_model_routers (aligning with rollout.py);
* --opd-type sglang -> vllm; engine_overrides rename;
* sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference,
wired to a real --vllm-enable-deterministic-inference flag (exports
VLLM_BATCH_INVARIANT=1);
* consistent_hash session-id routing uses vllm-router's x-session-id header;
* drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings.
- Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the
plugin-contract tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: finish de-SGLang + fold OPD/router-policy into runtime
- naming: replace residual generic "rollout engine"/"engine"/"comm" wording
with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text;
http_utils comments; rollout.py "inference workers"). sglang->vllm is correct,
sglang->generic is not.
- megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors
global. That was sglang-only: sglang's loader torch.cat's both shards within a
single load_weights call (needs them co-bucketed), whereas vLLM loads each shard
independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the
misleading "merge into single fused name" comment.
- docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the
build-time `import sglang` smoke check (slime no longer imports sglang_router).
- OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions
(prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs.
- routing replay: register --vllm-router-policy (dest=router_policy) so the
consistent_hash x-session-id session-affinity path is actually wired (was dead).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup
- vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate
`data["vllm"]` directly, restoring the "Accept both server_groups / legacy
engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line.
- command_utils.execute_train: re-add a process kill for leftover rollout engines
as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no
vLLM equivalent), so stale engines don't hold GPUs/ports across runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop test changes from runtime PR; tests live in the tests+CI PR (#40)
The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename
are coupled to the test/CI rename effort and are owned by #40. Restore them to
main here so #18 is purely the SGLang runtime removal. #18 merges first; #40
rebases and re-lands the vLLM test versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env
- fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations
of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers
(per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl
packer). deep_gemm is imported lazily inside the functions (as SGLang does), so
module import no longer requires deep_gemm. This fixes the module-level
`NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any
deep_gemm image, and drops the invented sf-stride fixup block that was not in
upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted
(is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper.
- vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 +
VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT,
replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env
now lives in the subprocess env builder (single source of truth).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs
- fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm,
consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the
module-level crash: the `try` block referenced `_get_tma_aligned_size` before it
was bound (the "pre-imported with fallback" import was never written), which
raised NameError whenever deep_gemm imported successfully -- and NameError is not
caught by `except ImportError`, so `import megatron_to_hf` crashed on any
deep_gemm image. Replace the bogus self-assignment with the real import:
`from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`.
- trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason
+ token usage to the vllm_inference_generate span (mirrors SGLang's
build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives
in vLLM's own OTLP traces).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(opd): score teacher via /inference/v1/generate with prompt_logprobs
Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto
vime's native /inference/v1/generate (the same endpoint the rollout engines
use), and fix three latent issues:
1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop
defaulting to args.hf_checkpoint (the *student* name, which mis-names a
teacher!=student server). Add --opd-teacher-model; send `model` only when
set, otherwise omit it (single-model teacher servers use their loaded model).
2. multimodal: the old code sent image_data to a token-only endpoint, which is
invalid. Raise NotImplementedError until the
/v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors
slime.rollout.vllm_rollout.generate).
3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert
it is present and length-aligned with token_ids, assert the per-sample tensor
covers response_length, and raise (not silently return 0.0) on a missing
token logprob. vLLM always includes the actual prompt token in
prompt_logprobs, so a miss is a real error.
Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]).
Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher
that exposes /inference/v1/generate and point --rm-url at it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(clean-sglang): purge SGLang from tools, train scripts, and build infra
tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword
profile/replay helpers to vLLM and map analyzer hints to vLLM flags
(--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments
SGLang -> vLLM.
build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300
sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale.
docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real
multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/
ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404
(CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already
CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker
selects the arm64 image automatically on an ARM host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image)
The single build-conda job ran `bash build_conda.sh` (removed in the previous
commit) inside an lmsysorg/sglang container. With the SGLang-only conda path
gone, the whole workflow is dead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments
docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of
the inherited sgl-project.github.io repo. .claude/skills/*: update the dead
`slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real
default is slime.rollout.vllm_rollout.generate_rollout).
justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it
equalled the Dockerfile default; the multi-arch manifest already resolves
arm64). train{,_async}.py: drop stray "the" in the W&B comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed
vLLM's set_process_title() renames the VRAM-holding subprocesses
(VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no
longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"`
matched only the launcher and left engine/worker children holding GPU
memory, leaking it into the next run — masked only by the indiscriminate
`pkill -9 python`, which is unsafe on colocate/shared nodes.
Match both the launcher and the renamed children with
`pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the
pattern from matching pkill's own cmdline. This makes the broad python
kill unnecessary, so its already-commented-out lines are removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(speculative): use method=mtp (not eagle) for embedded-MTP models
vLLM's SpeculativeConfig requires an explicit draft `model` for
method=eagle; with only num_speculative_tokens set it raises
"num_speculative_tokens was provided but without speculative model".
The migrated configs in scripts/examples/docs pass no model, so they must
use method=mtp, which reuses the target checkpoint's embedded MTP layer
(DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5).
The two docs examples that pass an explicit "model" are genuine eagle
usage and are left unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(vllm): launch each rollout engine with its ServerGroup's per-group TP
launch_server_process / _init_normal derived tensor-parallel size and
CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring
the per-engine num_gpus_per_engine already carried on the VLLMEngine actor.
A ServerGroup configured with num_gpus_per_engine greater than the global
flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync
rendezvous sized world_size from engine_gpu_counts (the per-group value).
The two disagreed: the trainer waited for a rank the under-sized engine
never started, so init_weight_transfer_engine hung for 300s
("3/4 clients joined") and the job failed.
Honor the per-engine num_gpus_per_engine at launch, falling back to the
global flag when unset (matches the SGLang path and PR #66's
_compute_server_args).
Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now
launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s
(was a 301s timeout), and rollout+eval proceed.
AI assistance (Claude Code) was used for this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout
vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_"
(never "vllm_router_"), and excludes host/port from its CLI via
exclude_host_port=True. So:
- --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does
not own these CLI flags, vime does (populated via _start_router's manual
router_args.host/port assignment), so the vllm_ prefix is free and marks them
as vime-owned endpoint config.
- --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it
is a genuine RouterArgs field, so it shares the --router-* namespace with
policy / cache_threshold / retries / … and flows through from_cli_args like
the other knobs.
- --vllm-router-policy keeps dest=router_policy (unchanged).
Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port,
which never matched the vllm_engine reader) and updates README/README_zh prose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
gcanlin
pushed a commit
that referenced
this pull request
May 30, 2026
… test (#77) The hybrid router-arg convention (settled in #18/#40) keeps the vllm_ prefix only for the endpoint-locating flags --vllm-router-ip and --vllm-router-port; every other vllm-router knob is passed bare with a --router- prefix (dest router_*). The glm4.7 PD/mooncake integration test still passed the pre-hybrid name --vllm-router-request-timeout-secs, which is no longer a registered flag, so the 1200s timeout never reached router_args.request_timeout_secs. This test is GPU-only and outside the unit suite, so the rename sweep in #40 missed it. Rename the flag to --router-request-timeout-secs to match arguments.py (dest router_request_timeout_secs) and rollout.py's consumer. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
momo609
pushed a commit
that referenced
this pull request
Jun 8, 2026
* [Clean] Remove SGLang runtime code
Rebuilt against current main so the PR contains only the SGLang runtime
removal -- the docs / tests-ci / examples / scripts / docker portions were
split into separate PRs that have since merged.
- Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py,
rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all
docker/**/sglang.patch files.
- Rename the rollout config module sglang_utils/sglang_config.py ->
vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config ->
_resolve_vllm_config, --sglang-config -> --vllm-config); inline the
GPU_MEMORY_TYPE_* constants in rollout.py.
- Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported
through the sglang shim; repoint quantizer_fp8 to it.
- Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead
sglang-router dependency from requirements.txt.
- Finish the SGLang->vLLM rename in the runtime so it is internally consistent and
matches the tests landing in the tests/CI PR:
* router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout);
* get_model_url reads vllm_model_routers (aligning with rollout.py);
* --opd-type sglang -> vllm; engine_overrides rename;
* sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference,
wired to a real --vllm-enable-deterministic-inference flag (exports
VLLM_BATCH_INVARIANT=1);
* consistent_hash session-id routing uses vllm-router's x-session-id header;
* drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings.
- Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the
plugin-contract tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: finish de-SGLang + fold OPD/router-policy into runtime
- naming: replace residual generic "rollout engine"/"engine"/"comm" wording
with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text;
http_utils comments; rollout.py "inference workers"). sglang->vllm is correct,
sglang->generic is not.
- megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors
global. That was sglang-only: sglang's loader torch.cat's both shards within a
single load_weights call (needs them co-bucketed), whereas vLLM loads each shard
independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the
misleading "merge into single fused name" comment.
- docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the
build-time `import sglang` smoke check (slime no longer imports sglang_router).
- OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions
(prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs.
- routing replay: register --vllm-router-policy (dest=router_policy) so the
consistent_hash x-session-id session-affinity path is actually wired (was dead).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup
- vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate
`data["vllm"]` directly, restoring the "Accept both server_groups / legacy
engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line.
- command_utils.execute_train: re-add a process kill for leftover rollout engines
as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no
vLLM equivalent), so stale engines don't hold GPUs/ports across runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop test changes from runtime PR; tests live in the tests+CI PR (#40)
The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename
are coupled to the test/CI rename effort and are owned by #40. Restore them to
main here so #18 is purely the SGLang runtime removal. #18 merges first; #40
rebases and re-lands the vLLM test versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env
- fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations
of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers
(per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl
packer). deep_gemm is imported lazily inside the functions (as SGLang does), so
module import no longer requires deep_gemm. This fixes the module-level
`NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any
deep_gemm image, and drops the invented sf-stride fixup block that was not in
upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted
(is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper.
- vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 +
VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT,
replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env
now lives in the subprocess env builder (single source of truth).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs
- fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm,
consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the
module-level crash: the `try` block referenced `_get_tma_aligned_size` before it
was bound (the "pre-imported with fallback" import was never written), which
raised NameError whenever deep_gemm imported successfully -- and NameError is not
caught by `except ImportError`, so `import megatron_to_hf` crashed on any
deep_gemm image. Replace the bogus self-assignment with the real import:
`from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`.
- trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason
+ token usage to the vllm_inference_generate span (mirrors SGLang's
build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives
in vLLM's own OTLP traces).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(opd): score teacher via /inference/v1/generate with prompt_logprobs
Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto
vime's native /inference/v1/generate (the same endpoint the rollout engines
use), and fix three latent issues:
1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop
defaulting to args.hf_checkpoint (the *student* name, which mis-names a
teacher!=student server). Add --opd-teacher-model; send `model` only when
set, otherwise omit it (single-model teacher servers use their loaded model).
2. multimodal: the old code sent image_data to a token-only endpoint, which is
invalid. Raise NotImplementedError until the
/v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors
slime.rollout.vllm_rollout.generate).
3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert
it is present and length-aligned with token_ids, assert the per-sample tensor
covers response_length, and raise (not silently return 0.0) on a missing
token logprob. vLLM always includes the actual prompt token in
prompt_logprobs, so a miss is a real error.
Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]).
Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher
that exposes /inference/v1/generate and point --rm-url at it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(clean-sglang): purge SGLang from tools, train scripts, and build infra
tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword
profile/replay helpers to vLLM and map analyzer hints to vLLM flags
(--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments
SGLang -> vLLM.
build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300
sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale.
docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real
multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/
ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404
(CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already
CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker
selects the arm64 image automatically on an ARM host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image)
The single build-conda job ran `bash build_conda.sh` (removed in the previous
commit) inside an lmsysorg/sglang container. With the SGLang-only conda path
gone, the whole workflow is dead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments
docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of
the inherited sgl-project.github.io repo. .claude/skills/*: update the dead
`slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real
default is slime.rollout.vllm_rollout.generate_rollout).
justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it
equalled the Dockerfile default; the multi-arch manifest already resolves
arm64). train{,_async}.py: drop stray "the" in the W&B comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed
vLLM's set_process_title() renames the VRAM-holding subprocesses
(VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no
longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"`
matched only the launcher and left engine/worker children holding GPU
memory, leaking it into the next run — masked only by the indiscriminate
`pkill -9 python`, which is unsafe on colocate/shared nodes.
Match both the launcher and the renamed children with
`pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the
pattern from matching pkill's own cmdline. This makes the broad python
kill unnecessary, so its already-commented-out lines are removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(speculative): use method=mtp (not eagle) for embedded-MTP models
vLLM's SpeculativeConfig requires an explicit draft `model` for
method=eagle; with only num_speculative_tokens set it raises
"num_speculative_tokens was provided but without speculative model".
The migrated configs in scripts/examples/docs pass no model, so they must
use method=mtp, which reuses the target checkpoint's embedded MTP layer
(DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5).
The two docs examples that pass an explicit "model" are genuine eagle
usage and are left unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(vllm): launch each rollout engine with its ServerGroup's per-group TP
launch_server_process / _init_normal derived tensor-parallel size and
CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring
the per-engine num_gpus_per_engine already carried on the VLLMEngine actor.
A ServerGroup configured with num_gpus_per_engine greater than the global
flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync
rendezvous sized world_size from engine_gpu_counts (the per-group value).
The two disagreed: the trainer waited for a rank the under-sized engine
never started, so init_weight_transfer_engine hung for 300s
("3/4 clients joined") and the job failed.
Honor the per-engine num_gpus_per_engine at launch, falling back to the
global flag when unset (matches the SGLang path and PR #66's
_compute_server_args).
Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now
launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s
(was a 301s timeout), and rollout+eval proceed.
AI assistance (Claude Code) was used for this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout
vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_"
(never "vllm_router_"), and excludes host/port from its CLI via
exclude_host_port=True. So:
- --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does
not own these CLI flags, vime does (populated via _start_router's manual
router_args.host/port assignment), so the vllm_ prefix is free and marks them
as vime-owned endpoint config.
- --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it
is a genuine RouterArgs field, so it shares the --router-* namespace with
policy / cache_threshold / retries / … and flows through from_cli_args like
the other knobs.
- --vllm-router-policy keeps dest=router_policy (unchanged).
Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port,
which never matched the vllm_engine reader) and updates README/README_zh prose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
momo609
pushed a commit
that referenced
this pull request
Jun 8, 2026
* tests + CI: complete sglang→vllm rename across tests/ and .github/ Split from PR #18 (gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / **tests+CI** / core runtime. 42 files / +~750 / -~700. These are bundled in a single PR because the CI workflows reference test file names by string — splitting them would create a window where either tests are renamed but CI still points at the old names, or vice versa, breaking CI mid-roll. What this PR does: (A) tests/ (38 files): - Mechanical CLI-flag rename: --sglang-* → --vllm-* equivalents in all test scripts (matches the table now used in scripts/ and examples/). - Variable rename: SGLANG_ARGS → VLLM_ARGS where present. - 4 file renames (R086-R091, all >85% similarity): test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py test_qwen2.5_0.5B_sglang_config.py → test_qwen2.5_0.5B_vllm_config.py test_qwen2.5_0.5B_sglang_config_distributed.py → test_qwen2.5_0.5B_vllm_config_distributed.py test_sglang_config_mixed_offload.py → test_vllm_config_mixed_offload.py test_sglang_config_mixed_offload_ft.py → test_vllm_config_mixed_offload_ft.py tests/utils/test_sglang_config.py → tests/utils/test_vllm_config.py - 2 new tests for the IPC weight-transfer path landed in PR #18: tests/test_update_weight_from_tensor.py tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py (These are PR #22 / colocate-IPC test coverage; the production code the slim PR #18 ships will rely on the same code from PR #22.) (B) .github/ (4 files): - workflows/conda-ci.yml: container image lmsysorg/sglang → vime (inferactinc/public:vime-vllm-cu129-latest). - workflows/pr-test.yml + pr-test.yml.j2 (template): * Container images (slimerl/slime[-test]:latest → vime image) on every job that ran on the sglang-era base. * e2e-test-sglang-config job → e2e-test-vllm-config job (renamed label `run-ci-sglang-config` → `run-ci-vllm-config`; matrix `test_file` entries updated to point at the renamed test files in (A)). * e2e-test-megatron + e2e-test-image matrices: `_opd_sglang.py` entries → `_opd_vllm.py`. - ISSUE_TEMPLATE/bug_report.yml: drop the "SGLang version (if relevant):" environment field, add "vLLM version:" and "vllm-router version:" lines. (PR #36 already changed "CUDA/ROCm version" → "CUDA version" earlier; that change is preserved.) Sgl residue intentionally kept (4 hits — all anti-regression assertions that prove sglang code paths are gone, not residual references to bring back): - tests/test_update_weight_from_tensor.py:753 — comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code". - tests/unit/backends/vllm_utils/test_arguments.py:233-237 — three assertions that --sglang-router-ip, --sglang-router-port, and sglang_router_ip are NOT present in the argument parser. Tests + CI must land together; splitting them risks a window where the CI matrix references test files by names that don't exist yet (or no longer exist). After this lands, the test_file string in CI matches the test files on disk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * on_policy_distillation: port from SGLang to vLLM /v1/completions Follow-up on the test rename in this PR: test_qwen2.5_0.5B_opd_sglang.py → test_qwen2.5_0.5B_opd_vllm.py. The test only spawns a vLLM teacher and exercises the OPD pipeline; the real broken piece was slime/rollout/on_policy_distillation.py, which PR #18 left in SGLang request/response shape: request fields: "max_new_tokens": 0 (vLLM: "max_tokens") "return_logprob": True (sglang-only) "logprob_start_len": 0 (sglang-only) response parsing: reward["meta_info"]["input_token_logprobs"] (sglang shape) vLLM 0.21 supports the same workflow natively via `prompt_logprobs`: request to POST /v1/completions: { "model": <teacher>, "prompt_token_ids": sample.tokens, "max_tokens": 1, "temperature": 0, "prompt_logprobs": 1, "logprobs": 0, "skip_special_tokens": False, } response: response["choices"][0]["prompt_logprobs"] # list[dict[int, Logprob] | None] where Logprob is {"logprob": float, "rank": int, "decoded_token": str} References checked against vllm source: - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:91 (request: prompt_logprobs: int | None) - reference/vllm/vllm/entrypoints/openai/completion/protocol.py:487 (response: prompt_logprobs: list[dict[int, Logprob] | None] | None) - reference/vllm/vllm/logprobs.py:13 (Logprob dataclass: logprob/rank/decoded_token) Implementation notes: 1. JSON serializes int dict keys as strings, so `_logprob_for_token` tries both `pos_entry.get(token_id)` and `pos_entry.get(str(token_id))`. 2. `pos_entry` is `None` at position 0 (no prior context) — handled explicitly. We also gracefully degrade if a token at position `i` is not in the top-1 logprob dict (falls back to 0.0, same as the prior sglang code would do). 3. The Logprob dataclass `decoded_token` field is unused; we only read `.logprob`. Both dict and `Logprob` shapes are accepted in case the server uses a flatter serialization toggle. 4. `args.opd_teacher_model` is the new model-name arg; falls back to `args.hf_checkpoint` if not set, mirroring how vime's other rollout paths derive the model name. Smoke-tested `_logprob_for_token` locally: - None entry → 0.0 - int key + dict value → logprob - str key (JSON shape) → logprob - missing token → 0.0 - flattened float value → float Also drops 3 lines from tests/unit/backends/vllm_utils/test_arguments.py: the `--sglang-router-ip`/`--sglang-router-port`/`sglang_router_ip` anti- regression assertions. Once the slim PR #18 lands and sglang is gone from the runtime, those assertions are vacuous; treating sglang as non-existent per the project policy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com> * tests: drop duplicate smoke test updates from PR40 * test(update_weight_from_tensor): drop stale _apply_monkey_patch_torch_reductions patch The inner ``with patch(f"{MODULE_PATH}._apply_monkey_patch_torch_reductions"):`` context in _run_update suppressed a helper call that PR #48 has since deleted from update_weight_from_tensor.py (commit 39bf899 on aoshen/align-ipc-rpc-with-slime). After that PR lands the patched attribute won't exist and this line raises AttributeError. Remove it now so the test survives PR #48 merge. The ``sglang_mod.monkey_patch_torch_reductions = MagicMock()`` stub on the fake sglang module is intentionally kept: on this branch the production code still imports it via ``from ..sglang import monkey_patch_torch_reductions`` (both update_weight_from_tensor._apply_monkey_patch_torch_reductions on PR #40's view of main, and hf_weight_iterator_direct.py at module level). Removing the stub here would break the test on PR #40 alone; it can be dropped in a follow-up once PR #48 finishes removing every import site. Tests: ``tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py`` all 6 pass with this change applied to gcl/pr18-tests-ci HEAD. * tests: drop duplicate top-level test_update_weight_from_tensor.py The 786-line tests/test_update_weight_from_tensor.py is a stale rebase leftover from the original PR #18 branch — it predates the IPC test file PR #22 landed at the canonical unit-test path (tests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py) and predates PR #48's single-RPC weight-version contract. Comparing the two: * Both stub sys.modules / torch.distributed at module import time, so having two files compounds the test-isolation issue Gemini raised (PR #40 comment #1). * Coverage overlaps materially (e.g. test_ipc_init_called_on_first_update_only ≈ test_ipc_init_runs_once — same invariant, different wording). * The nested file is up-to-date with PR #48's RPC contract (update_weights_from_tensor.remote(**fields, weight_version=...)); the top-level file still uses the pre-#48 lifecycle shape and does not exercise the coordinator slot fields. * The nested path matches repo convention: tests/unit/ for mock-only unit tests, tests/ top level for e2e scripts. Closes Gemini comment #1 on PR #40. Gemini comment #2 (the same stub pattern in the surviving nested file) is a pre-existing issue from PR #22 / #48 and out of scope for this rename PR — to be addressed in a follow-up that converts _install_stubs() to an autouse module-scoped fixture with save/restore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(vllm_config): use real get_model_url default endpoint /inference/v1/generate get_model_url defaults to /inference/v1/generate (PR #18), not /v1/completions. Aligns this test with PR #18's test_vllm_config.py so the two PRs no longer conflict on this file and the assertion matches the actual runtime default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop on_policy_distillation.py from tests+CI PR (now owned by runtime PR #18) The OPD vLLM /v1/completions migration is a runtime change; it was folded into the core-runtime PR (#18). Restore this file to main here so the two PRs no longer overlap on it. #18 merges first, so this lands via #18. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop unit-test files now owned by runtime PR #18 test_vllm_config.py + the plugin_contracts tests are coupled to #18's runtime rename (they import vllm_config / vllm_rollout, which #18 creates). They live in #18; remove them here so the two PRs don't overlap. #18 merges first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: restore vLLM rollout args dropped during sglang→vllm rename The mechanical sglang→vllm rename dropped several rollout knobs instead of mapping them to their vLLM equivalents, weakening CI coverage (cuda-graph capture caps, speculative decoding, expert parallel). Restore them using the mapping established by the converted production scripts on main (run-glm4.7-30B-A3B.sh / run-glm5-744B-A40B.sh), verified against vLLM AsyncEngineArgs: --sglang-cuda-graph-max-bs N -> --vllm-max-cudagraph-capture-size N --sglang-cuda-graph-bs a b c -> --vllm-cudagraph-capture-sizes a b c --sglang-ep-size N -> --vllm-enable-expert-parallel --sglang-speculative-* (eagle) -> --vllm-speculative-config '{"method":"eagle","num_speculative_tokens":K}' Also: - glm4.7 pd: fix --vllm-max-num-seqs (was 8, taken from cuda-graph-max-bs; --sglang-max-running-requests was 16) and split out cuda-graph capture. - fix sglang→rollout mis-renames in temp-file prefixes (→ vllm_*). - test_vllm_config: rename test_update_weights_default_true → test_update_weights_defaults_to_none (it asserts `is None`). Dropped sglang flags with no vLLM equivalent (enable-dp-lm-head, moe-dense-tp-size, watchdog-timeout, mamba-scheduler-strategy, disaggregation-transfer-backend, enable-metrics) stay dropped; PD KV-transfer is driven by --prefill-num-servers + the --vllm-config prefill/decode topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(plugin_contracts): migrate from sglang_rollout to vllm_rollout The three plugin-contract tests still imported slime.rollout.sglang_rollout and called install_stubs(with_sglang_router=True), but _shared.install_stubs already dropped that parameter — so all three failed at collection (TypeError: unexpected keyword 'with_sglang_router'). Complete the migration: - install_stubs(with_sglang_router=True, ...) -> install_stubs(...) - import generate_and_rm / generate_rollout from slime.rollout.vllm_rollout - default rollout/eval path string -> slime.rollout.vllm_rollout.generate_rollout (matches runtime default at slime/utils/arguments.py:233) - FakeGenerateState: sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, with group_sampling_seeds defaulting to None and gated on the flag (mirrors the already-migrated tests/unit/rollout/test_vllm_rollout.py). All 34 plugin-contract cases pass (were 3 collection errors before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(update_weight_from_tensor): drop stale slime…megatron_utils.sglang mock The test pre-registered a sys.modules mock for slime.backends.megatron_utils.sglang (monkey_patch_torch_reductions), left over from when update_weight_from_tensor imported it. The module under test no longer imports that module (its real deps are get_gloo_group / HfWeightIteratorBase / update_weight_from_distributed), so the mock is dead. Removing it makes tests/ and .github/ fully sglang-free. Test still passes (7/7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Clean] Remove SGLang runtime code Rebuilt against current main so the PR contains only the SGLang runtime removal -- the docs / tests-ci / examples / scripts / docker portions were split into separate PRs that have since merged. - Delete dead SGLang server/runtime code: sglang_utils/{arguments,sglang_engine}.py, rollout/sglang_rollout.py, the megatron_utils/sglang.py re-export shim, and all docker/**/sglang.patch files. - Rename the rollout config module sglang_utils/sglang_config.py -> vllm_utils/vllm_config.py (SglangConfig -> VllmConfig, _resolve_sglang_config -> _resolve_vllm_config, --sglang-config -> --vllm-config); inline the GPU_MEMORY_TYPE_* constants in rollout.py. - Add megatron_utils/fp8_helpers.py for the UE8M0 fp8 helpers formerly re-exported through the sglang shim; repoint quantizer_fp8 to it. - Swap sglang_router -> vllm_router in http_utils/wandb_utils; drop the dead sglang-router dependency from requirements.txt. - Finish the SGLang->vLLM rename in the runtime so it is internally consistent and matches the tests landing in the tests/CI PR: * router args --router-* -> --vllm-router-* (vllm_router_ip/port/timeout); * get_model_url reads vllm_model_routers (aligning with rollout.py); * --opd-type sglang -> vllm; engine_overrides rename; * sglang_enable_deterministic_inference -> vllm_enable_deterministic_inference, wired to a real --vllm-enable-deterministic-inference flag (exports VLLM_BATCH_INVARIANT=1); * consistent_hash session-id routing uses vllm-router's x-session-id header; * drop dead trace helper build_sglang_meta_trace_attrs; de-SGLang comments/docstrings. - Rename test_sglang_config.py -> test_vllm_config.py and de-SGLang the plugin-contract tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: finish de-SGLang + fold OPD/router-policy into runtime - naming: replace residual generic "rollout engine"/"engine"/"comm" wording with concrete vLLM (engine_overrides -> vllm_overrides; arguments help text; http_utils comments; rollout.py "inference workers"). sglang->vllm is correct, sglang->generic is not. - megatron_to_hf: drop the q_a_proj/kv_a_proj_with_mqa pairing + _cached_tensors global. That was sglang-only: sglang's loader torch.cat's both shards within a single load_weights call (needs them co-bucketed), whereas vLLM loads each shard independently via stacked_params_mapping into fused_qkv_a_proj. Also fix the misleading "merge into single fused name" comment. - docker/Dockerfile: remove now-dead sglang/sglang-router --no-deps stubs + the build-time `import sglang` smoke check (slime no longer imports sglang_router). - OPD: migrate on_policy_distillation.py teacher logprobs to vLLM /v1/completions (prompt_logprobs) instead of sglang return_logprob / meta_info.input_token_logprobs. - routing replay: register --vllm-router-policy (dest=router_policy) so the consistent_hash x-session-id session-affinity path is actually wired (was dead). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Review follow-ups: mirror slime vllm_config parsing + restore vLLM process cleanup - vllm_config.from_yaml: drop the needless `models_raw` intermediate and iterate `data["vllm"]` directly, restoring the "Accept both server_groups / legacy engine_groups" comment -- mirrors slime's sglang_config.from_yaml line-for-line. - command_utils.execute_train: re-add a process kill for leftover rollout engines as `pkill -9 -f "vllm serve"` (the old `pkill -9 sglang` was dropped with no vLLM equivalent), so stale engines don't hold GPUs/ports across runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop test changes from runtime PR; tests live in the tests+CI PR (#40) The plugin_contracts tests and the test_sglang_config -> test_vllm_config rename are coupled to the test/CI rename effort and are owned by #40. Restore them to main here so #18 is purely the SGLang runtime removal. #18 merges first; #40 rebases and re-lands the vLLM test versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: copy SGLang verbatim (fix import crash); pin vLLM deep_gemm env - fp8_helpers.py: replace the bespoke rewrite with SGLang's exact implementations of quant_weight_ue8m0 / transform_scale_ue8m0 and their DeepGEMM helpers (per_block_cast_to_fp8, ceil_to_ue8m0, ceil_div, ceil_align, the torch-impl packer). deep_gemm is imported lazily inside the functions (as SGLang does), so module import no longer requires deep_gemm. This fixes the module-level `NameError: _get_tma_aligned_size` that crashed `import megatron_to_hf` on any deep_gemm image, and drops the invented sf-stride fixup block that was not in upstream. Only should_deepgemm_weight_requant_ue8m0 stays vLLM-adapted (is_deep_gemm_e8m0_used) since SGLang's reads SGLang-internal deep_gemm_wrapper. - vllm_engine.launch_server_process: set VLLM_USE_DEEP_GEMM=1 + VLLM_DEEP_GEMM_WARMUP=relax explicitly (setdefault) alongside VLLM_BATCH_INVARIANT, replacing SGLang's removed deep_gemm precompile/warmup envs. All vLLM engine env now lives in the subprocess env builder (single source of truth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fp8_helpers: revert to vLLM impl + fix the import NameError; add vLLM trace attrs - fp8_helpers.py: keep the vLLM-based implementation (uses vllm.utils.deep_gemm, consistent with the vLLM runtime) rather than the SGLang verbatim copy. Fix the module-level crash: the `try` block referenced `_get_tma_aligned_size` before it was bound (the "pre-imported with fallback" import was never written), which raised NameError whenever deep_gemm imported successfully -- and NameError is not caught by `except ImportError`, so `import megatron_to_hf` crashed on any deep_gemm image. Replace the bogus self-assignment with the real import: `from vllm.utils.deep_gemm import get_tma_aligned_size as _get_tma_aligned_size`. - trace_utils/vllm_rollout: add build_vllm_meta_trace_attrs and attach finish_reason + token usage to the vllm_inference_generate span (mirrors SGLang's build_sglang_meta_trace_attrs; vLLM responses lack the pd_* timing, which lives in vLLM's own OTLP traces). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(opd): score teacher via /inference/v1/generate with prompt_logprobs Move the vllm OPD teacher path off the OpenAI /v1/completions endpoint onto vime's native /inference/v1/generate (the same endpoint the rollout engines use), and fix three latent issues: 1. model field: /inference/v1/generate takes `model` as OPTIONAL. Stop defaulting to args.hf_checkpoint (the *student* name, which mis-names a teacher!=student server). Add --opd-teacher-model; send `model` only when set, otherwise omit it (single-model teacher servers use their loaded model). 2. multimodal: the old code sent image_data to a token-only endpoint, which is invalid. Raise NotImplementedError until the /v1/chat/completions/render -> /inference/v1/generate flow is wired (mirrors slime.rollout.vllm_rollout.generate). 3. logprob robustness: read top-level GenerateResponse.prompt_logprobs, assert it is present and length-aligned with token_ids, assert the per-sample tensor covers response_length, and raise (not silently return 0.0) on a missing token logprob. vLLM always includes the actual prompt token in prompt_logprobs, so a miss is a real error. Alignment is unchanged (plp[i] <-> tokens[i], skip pos 0, take [-response_length:]). Follow-up (separate, in the tests PR): the OPD e2e test must launch a teacher that exposes /inference/v1/generate and point --rm-url at it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): purge SGLang from tools, train scripts, and build infra tools/: drop dead `args.sglang_enable_ep_moe` shim (read nowhere); reword profile/replay helpers to vLLM and map analyzer hints to vLLM flags (--enforce-eager, --gpu-memory-utilization). train{,_async}.py: comments SGLang -> vLLM. build infra: remove build_conda.sh (SGLang-only conda path); drop the GB300 sgl-kernel install from the Dockerfile; delete docker/npu_patch/ wholesale. docker base image: bump to vLLM v0.22.0. justfile ARM recipes now pin the real multi-arch vLLM base images instead of the dead SGLANG_IMAGE_TAG/ ENABLE_SGLANG_PATCH build-args -- cu129-arm64 -> v0.22.0-cu129-ubuntu2404 (CUDA 12.9), cu13-arm64 -> v0.22.0-ubuntu2404 (the default-CUDA tag is already CUDA 13.0) + ENABLE_CUDA_13=1. vLLM tags are multi-arch manifests, so docker selects the arm64 image automatically on an ARM host. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(clean-sglang): drop conda-build workflow (ran deleted build_conda.sh on SGLang image) The single build-conda job ran `bash build_conda.sh` (removed in the previous commit) inside an lmsysorg/sglang container. With the SGLang-only conda path gone, the whole workflow is dead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(clean-sglang): fix stale SGLang refs in docs/skills; tidy comments docs/conf.py: point the "edit on GitHub" links at vllm-project/vime instead of the inherited sgl-project.github.io repo. .claude/skills/*: update the dead `slime/rollout/sglang_rollout.py` references to `vllm_rollout.py` (the real default is slime.rollout.vllm_rollout.generate_rollout). justfile: drop the redundant BASE_IMAGE override on release-cu129-arm64 (it equalled the Dockerfile default; the multi-arch manifest already resolves arm64). train{,_async}.py: drop stray "the" in the W&B comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): use method=mtp (not eagle) in vllm speculative config The migrated speculative configs pass no draft `model`, so method=eagle raises "num_speculative_tokens was provided but without speculative model" in vLLM's SpeculativeConfig. These models carry embedded MTP layers, so method=mtp is correct and unblocks the mimo MTP-only-grad test (#19). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cleanup): target renamed vLLM subprocesses in pkill so VRAM is freed vLLM's set_process_title() renames the VRAM-holding subprocesses (VLLM::EngineCore, VLLM::Worker_TP*, vllm::router), so their cmdline no longer contains "vllm serve". The previous `pkill -9 -f "vllm serve"` matched only the launcher and left engine/worker children holding GPU memory, leaking it into the next run — masked only by the indiscriminate `pkill -9 python`, which is unsafe on colocate/shared nodes. Match both the launcher and the renamed children with `pkill -9 -f '[v]llm serve|VLL[M]::'`; the [v]/[M] bracket trick keeps the pattern from matching pkill's own cmdline. This makes the broad python kill unnecessary, so its already-commented-out lines are removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(speculative): use method=mtp (not eagle) for embedded-MTP models vLLM's SpeculativeConfig requires an explicit draft `model` for method=eagle; with only num_speculative_tokens set it raises "num_speculative_tokens was provided but without speculative model". The migrated configs in scripts/examples/docs pass no model, so they must use method=mtp, which reuses the target checkpoint's embedded MTP layer (DeepSeek-R1, GLM-4.x-MoE, MiMo, Qwen3-Next/3.5). The two docs examples that pass an explicit "model" are genuine eagle usage and are left unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(vllm): launch each rollout engine with its ServerGroup's per-group TP launch_server_process / _init_normal derived tensor-parallel size and CUDA_VISIBLE_DEVICES from the global --rollout-num-gpus-per-engine, ignoring the per-engine num_gpus_per_engine already carried on the VLLMEngine actor. A ServerGroup configured with num_gpus_per_engine greater than the global flag (e.g. tp=2) therefore launched as tp=1, while the NCCL weight-sync rendezvous sized world_size from engine_gpu_counts (the per-group value). The two disagreed: the trainer waited for a rank the under-sized engine never started, so init_weight_transfer_engine hung for 300s ("3/4 clients joined") and the job failed. Honor the per-engine num_gpus_per_engine at launch, falling back to the global flag when unset (matches the SGLang path and PR #66's _compute_server_args). Verified on H200: tests/test_qwen2.5_0.5B_vllm_config_distributed now launches engine0 tp=2 / engine1 tp=1, update_weights completes in 1.1s (was a 301s timeout), and rollout+eval proceed. AI assistance (Claude Code) was used for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ckpt): add --dist-ckpt-optim-fully-reshardable for PAO+offload save/load test_qwen3_4B_ckpt.py uses precision-aware optimizer + cpu-offload (HybridDeviceOptimizer). Under the default dp_reshardable (bucket-centric) optimizer sharding, save/load produce unequal-length param_state lists, so dist-ckpt load fails with "Cannot merge two lists with different lengths (81 and 79)". fully_reshardable is model-centric and immune to bucket-layout changes. Verified on the r3 image (Megatron-LM 0.16.0rc0 @ 1dcf0da): save+load both succeed, and source review confirms master_param / step / HybridDeviceOptimizer sync are handled on this path. This is the flag described in PR #50 that was never actually merged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(router-args): hybrid naming — vllm_ for ip/port, bare router_ for timeout vllm-router's RouterArgs.from_cli_args only supports prefix "" or "router_" (never "vllm_router_"), and excludes host/port from its CLI via exclude_host_port=True. So: - --vllm-router-ip / --vllm-router-port keep the vllm_ prefix: RouterArgs does not own these CLI flags, vime does (populated via _start_router's manual router_args.host/port assignment), so the vllm_ prefix is free and marks them as vime-owned endpoint config. - --router-request-timeout-secs goes bare (dest router_request_timeout_secs): it is a genuine RouterArgs field, so it shares the --router-* namespace with policy / cache_threshold / retries / … and flows through from_cli_args like the other knobs. - --vllm-router-policy keeps dest=router_policy (unchanged). Also fixes conftest fixture to seed vllm_router_ip/port (was bare router_ip/port, which never matched the vllm_engine reader) and updates README/README_zh prose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
momo609
pushed a commit
that referenced
this pull request
Jun 8, 2026
… test (#77) The hybrid router-arg convention (settled in #18/#40) keeps the vllm_ prefix only for the endpoint-locating flags --vllm-router-ip and --vllm-router-port; every other vllm-router knob is passed bare with a --router- prefix (dest router_*). The glm4.7 PD/mooncake integration test still passed the pre-hybrid name --vllm-router-request-timeout-secs, which is no longer a registered flag, so the 1200s timeout never reached router_args.request_timeout_secs. This test is GPU-only and outside the unit suite, so the rename sweep in #40 missed it. Rename the flag to --router-request-timeout-secs to match arguments.py (dest router_request_timeout_secs) and rollout.py's consumer. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split from PR #18 (
gcl/clean-sglang). One of 4 PRs splitting the original PR #18 by content area: docs (#38) / examples (#39) / tests+CI / core runtime.42 files.
Why tests + CI bundled in one PR
The CI workflows in
.github/workflows/pr-test.ymlreference test file names by string in the matrix:PR #18 renames those tests to
_vllm_config.py. If we split:Bundled → no broken window.
What this PR does
(A)
tests/(38 files)Mechanical CLI-flag rename:
--sglang-*→--vllm-*equivalents (same table as PR examples/scripts: complete sglang→vllm rename + drop SGLang-only examples #39).Variable rename:
SGLANG_ARGS→VLLM_ARGSwhere present.4 file renames (all >85% similarity):
test_qwen2.5_0.5B_opd_sglang.pytest_qwen2.5_0.5B_opd_vllm.pytest_qwen2.5_0.5B_sglang_config.pytest_qwen2.5_0.5B_vllm_config.pytest_qwen2.5_0.5B_sglang_config_distributed.pytest_qwen2.5_0.5B_vllm_config_distributed.pytest_sglang_config_mixed_offload.pytest_vllm_config_mixed_offload.pytest_sglang_config_mixed_offload_ft.pytest_vllm_config_mixed_offload_ft.pytests/utils/test_sglang_config.pytests/utils/test_vllm_config.py2 new tests for the IPC weight-transfer path (PR [feat] Support Colocated Weight Sync via CUDA IPC for vime #22 territory):
tests/test_update_weight_from_tensor.pytests/unit/backends/megatron_utils/update_weight/test_update_weight_from_tensor.py(B)
.github/(4 files)workflows/conda-ci.yml:lmsysorg/sglang:v0.5.0rc0-cu126→inferactinc/public:vime-vllm-cu129-latest.workflows/pr-test.yml+pr-test.yml.j2:slimerl/slime[-test]:latest→ vime image.e2e-test-sglang-configjob →e2e-test-vllm-config(label + matrix updated to renamed test files).e2e-test-megatron+e2e-test-imagematrices:_opd_sglang.py→_opd_vllm.py.ISSUE_TEMPLATE/bug_report.yml: dropSGLang version (if relevant):env field, addvLLM version:andvllm-router version:lines (CUDA/ROCm→CUDAchange from docker/scripts/runtime: remove AMD / ROCm platform support #36 preserved).Sgl residue intentionally kept (4)
All are anti-regression assertions confirming sglang code paths are gone, not residual references:
tests/test_update_weight_from_tensor.py:753— comment "The vLLM IPC implementation must NOT contain sglang-style Gloo gather code".tests/unit/backends/vllm_utils/test_arguments.py:233-237— 3 assertions that--sglang-router-ip,--sglang-router-port, andsglang_router_ipare NOT present in the argument parser.Test plan
git grep -iE 'sglang' .github/ tests/returns only the 4 anti-regression assertion lines abovetest_filestrings match actual files on disk:for f in $(grep -oE 'test_[a-zA-Z0-9._]+\.py' .github/workflows/pr-test.yml | sort -u); do test -f "tests/$f" || echo "MISSING: $f"; done🤖 Generated with Claude Code
Co-authors
Co-authored-by: Canlin Guo canlinguosdu@gmail.com
Test execution status (h200-0 + h200-1, 16 × H200, 2026-05-26 → 05-27)
Each test was run inside
inferactinc/public:vime-vllm-cu129-latest(PR #48 base + 2 local patches; see PR #48 description) with--num-rollout 2and--use-wandbstripped. r3 tests use a separately builtvime-vllm-r3:testimage (vllm nightlycu129-nightly-ff712f6447093d07747c88680b9d006b119f5890, which carries PR #39568).Legend: ✅ pass · ❌ real bug · ⏭️ blocked on infra/setup · ⏸️ not run · 🔧 needs local patch to pass
tests/(e2e)✅
test_qwen2.5_0.5B_short.py— 4 GPU, basic smoke (PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48 base + patches)✅
test_qwen2.5_0.5B_async_short.py— 4 GPU, async rollout. 🔧 needsos.environ.pop("http_proxy"/"HTTP_PROXY"/...)→ add, Nonedefault (4 calls); raisesKeyErroron hosts where the env var is unset. Trivial test fix.✅
test_qwen2.5_0.5B_ppo_critic_only_short.py— 4 GPU✅
test_qwen3.5_0.8B_gsm8k_short.py— 4 GPU✅
test_qwen3.5_0.8B_gsm8k_async_short.py— 4 GPU. 🔧 sameos.environ.popfix as above.✅
test_qwen2.5_0.5B_debug_rollout_then_train.py— 8 GPU, debug rollout + train two-phase✅
test_qwen3_0.6B_parallel_check.py— 8 GPU. Sweeps tp ∈ {1,2,4,8} × pp ∈ {1,2,4} × cp ∈ {1,2,4,8} fornum_gpus ∈ {8,4,2}. Required the 2 PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48 follow-up patches below; ran the full 2-hour sweep without leader/UUID errors after the patches.✅
test_qwen3_0.6B_parallel_check.pyexposed two regressions in PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48; both have fixes drafted:mpu.get_tensor_model_parallel_rank() == 0→rank == start(slot leader). Megatron-TP=1 + rollout-TP>1 made every slot rank a coordinator →start_weight_update called while a weight update is already active.mpu.get_tensor_model_parallel_group()→ per-slotdist.new_group(ranks=[start..end)). When Megatron-TP != rollout-TP, the Megatron-TP group does not cover the engine slot; merged dict missed peer UUIDs →IPC handle not found for GPU UUID ....✅
test_qwen3_4B_ppo.py— 8 GPU PPO + critic colocate, IPC heavy✅
test_qwen3_4B_ppo_train_critic_only.py— 8 GPU✅
test_qwen3_4B_ppo_disaggregate.py— 8 GPU rollout-disaggregated. First attempt hit a Ray transient (No available agent to submit job); retry passed.✅
test_mimo_7B_mtp_only_grad.py— 8 GPU, MTP gradient path✅
test_moonlight_16B_A3B.py— 8 GPU MoE colocate, heavy IPC✅
test_quick_start_glm4_9B.py— 8 GPU GLM smoke❌
test_qwen3_4B_ckpt.py— 8 GPU.ValueError: Cannot merge two lists with different lengths (81 and 52, encountered at level ('optimizer', 'param_state', 0, ...))on the load step. Reproduces on both the old PR [Clean] Remove SGlang runtime code #18 branch and PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48 base — not a PR [Clean] Remove SGlang runtime code #18/refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48 regression. Looks like Megatron-LMdist_checkpointingcannot reload an optimizer state saved under a different TP/PP shape. Needs investigation inslime/backends/megatron_utils/checkpoint.py(does it serialize the shape on save?). No fix PR yet — orthogonal to the IPC RPC work.❌
test_qwen3_30B_A3B.py— 8 GPU.ValueError: The output_size of gate's and up's weight = 96 is not divisible by weight quantization block_n = 128.Qwen3-30B-A3B-FP8.moe_intermediate_size=768; TP=8 gives 96 per rank. Test config bug, not slime. Use TP ∈ {1, 2, 3, 6} so768/TPstays divisible by 128. No fix PR yet (one-line--tensor-model-parallel-sizechange in the test).⏭️
test_glm4.7_30B_A3B_pd_mooncake.py— needs Mooncake KV-store infra deployed on host + GLM-4.7-Flash download (HF repo name not confirmed). Also note:--actor-num-nodes 1despite the "dual node" name; this is single-node PD-disagg, not 2-node Ray.⏭️
test_qwen3.6_35B_A3B_pd_mooncake.py— same Mooncake blocker. Model itself is on disk (models-shared/Qwen3.6-35B-A3B).⏭️
test_qwen3_30B_A3B_r3.py— needs vllm-nightly image (built locally asvime-vllm-r3:test) but also blocked by FP8 block_n config bug above.🔧
test_moonlight_16B_A3B_r3.py— 8 GPU. Tried with the new r3 image. Two import bugs surface on the r3 image:slime/ray/rollout.py:16—from sglang.srt.constants import GPU_MEMORY_TYPE_*. Easy patch: inline the 3 string constants ("cuda_graph"/"kv_cache"/"weights").slime/backends/megatron_utils/sglang.py:11,13—from sglang.srt.utils.patch_torch import monkey_patch_torch_reductions(referenced viasglang.monkey_patch_torch_reductionsinupdate_weight_from_tensor.py). Whole file plus its callers must be ported. Not blocking on the old image (sglang 0.5.10 is preinstalled there); only surfaces on the rebuilt nightly r3 image where sglang is absent. No fix PR yet — wider cleanup task.⏸️
test_gspo.sh— never executed.tests/(unit / non-e2e, at thetests/root)These 6 files live at
tests/root but are pytest-style unit tests, not Ray-job e2e. Run as part of Phase 1 on the PR #18 branch.test_chunked_gae.pytest_megatron_argument_validation.pytest_qwen3_5_mtp_bridge_mapping.pytest_qwen3_linear_attention_cu_seqlens.pytest_update_weight_from_tensor.py— pytest harness around the IPC weight transfer; distinct from the same-named module underslime/backends/megatron_utils/update_weight/.test_vllm_generate_endpoint.pytests/files that are renamed-from-sglang by this PR (cannot run on base)These don't exist on
origin/main; they only show up in PR #48 base after PR #48 (which is built on this PR). One of them caught a real PR #18 NCCL bug when run on the oldervime-pr18-pushbranch:test_qwen2.5_0.5B_opd_vllm.py— N/A onorigin/main. Will run after this PR merges.test_qwen2.5_0.5B_vllm_config.py— N/A onorigin/main. Will run after this PR merges.test_qwen2.5_0.5B_vllm_config_distributed.py— Real bug on the PR [Clean] Remove SGlang runtime code #18 branch. 4-GPU non-colocate distributed path:torch.distributed.DistStoreError: Timed out, 3/4 clients joined. The NCCL weight transfer rendezvous fails because only 3 of 4 expected clients connect. PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48's IPC patches don't touch this path (it goes throughupdate_weights_from_distributed, notupdate_weights_from_tensor). No fix PR yet — needs investigation in the distributed bridge connect/disconnect logic; likely a rank-count mismatch between trainer and rollout sides when not all rollout engines are colocated.test_vllm_config_mixed_offload.py— N/A onorigin/main. Will run after this PR merges.test_vllm_config_mixed_offload_ft.py— N/A onorigin/main. Will run after this PR merges.tests/unit/,tests/utils/,tests/plugin_contracts/(standalone)Ran on PR #18 branch (HEAD
8e03368) as Phase 1, 15 files: all pass except the 8 pre-existing failures intests/unit/backends/vllm_utils/test_vllm_engine.pyalready documented in PR #48's commit message (_weight_transfer_http_timeout,_response_json_or_fallback,server_host, ...). Those 8 are unchanged by PR #48 and not in the IPC RPC scope —main: 8 failed / 24 passed; with PR #48: 8 failed / 26 passed (the 2 new IPC RPC unit tests).Failures requiring no fix PR
test_qwen3_4B_ckpt.pydist_checkpointing.mergeacross TP shapestest_qwen3_30B_A3B.pyblock_n=128andmoe_intermediate_size=768--tensor-model-parallel-sizein the testTest bugs surfaced and fixes drafted
tests/test_qwen2.5_0.5B_async_short.py:117andtests/test_qwen3.5_0.8B_gsm8k_async_short.pyos.environ.pop("http_proxy")→os.environ.pop("http_proxy", None)(× 4 variants: lower/upperhttp_proxy/https_proxy)Open follow-up PRs against PR #48
The 2 IPC regressions caught by
test_qwen3_0.6B_parallel_check.pyneed to land before PR #48 merges:slime/backends/megatron_utils/update_weight/update_weight_from_tensor.py:connect_rollout_engines: build adist.new_group(ranks=range(start, end))per engine slot collectively across all trainer ranks, store inself._ipc_slot_group; replace coordinator conditionmpu.get_tensor_model_parallel_rank() == 0withrank == start._send_hf_chunk_via_ipc: replacetp_group/tp_sizeuses with the slot group/size.Test environment notes (not in this PR, applied at the harness layer)
SameFileErrorinhfdownloads (vllm's_hf_hub_download_to_local_dircallsshutil.copyfile).patch_hf_skip_v2.shshort-circuitshf download <repo> --local-dir <dst>whendst/config.jsonalready exists.patch_example_paths.shrewrites/root/<MODEL>to/root/models/<MODEL>and/root/<DATASET>to/root/datasets/<DATASET>for example scripts (tests already use the prefixed paths).Update 2026-05-28 — additional r3 image validation (
vime-vllm-r3:test+ PR #48 + PR #55)Ran another sweep of
tests/test_*.pyonvime-vllm-r3:test(vllm nightly w/ PR #39568) on top of PR #48 (3f93600..fd12344) merged onorigin/main+ PR #55 IPC destructure (_, ipc_args = reduce_tensor(weight)). Local source patches applied to master worktree:slime/ray/rollout.py:16— inlined 3 string constants forGPU_MEMORY_TYPE_*(dropsfrom sglang.srt.constants import ...)slime/backends/megatron_utils/sglang.py— every sglang import wrapped intry/except;monkey_patch_torch_reductionsfalls back to no-op when sglang isn't installedStatus changes vs the section above
test_moonlight_16B_A3B_r3.pymoonlight-long-r3long-run also PASS.test_glm4.7_30B_A3B_pd_mooncake.py--num-rollout 2); Mooncake KV store is referenced via sglang disagg backend args but is not strictly required when the test is colocated. Jobraysubmit_SpHjTuBLMiFBybRisucceeded.test_qwen3_30B_A3B_r3.pyactor.trainontrain_rollout_logprob_abs_diff/grad_normpath. Block reason changed from infra → R3-path-specific.test_qwen3_4B_ckpt.pyCannot merge 81 vs 52Cannot merge 81 vs 79test_qwen3_0.6B_parallel_check.pyAssertionError: grad norm mismatch: 0.502 != 0.422model.py:795fails. Worth investigating whether R3 routing replay introduces non-determinism that breaks parallel_check.New PASS entries (not in the table above)
test_qwen3_4B_ppo.pylong-run — 15 rollouts on r3, IPC weight sync stable end-to-endtest_moonlight_16B_A3B.pylong-run — 10 rollouts on r3, MoE IPC stabletest_mimo_7B_mtp_only_grad.pylong-run — 10 rollouts on r3, MTP path stabletest_qwen3_4B_ppo_train_critic_only.py— 8 GPU on r3 (PR tests + CI: complete sglang→vllm rename across tests/ and .github/ #40 section above had it as ✅ on PR refactor(weight-sync): align IPC RPC contract with slime (single-RPC version-with-data) #48 base, confirmed again on r3)r3 image systemic incompatibilities (new finding)
The
vime-vllm-r3:testimage is not a drop-in replacement forinferactinc/public:vime-vllm-cu129-latest. Tests that pass on cu129 fail on r3 because:megatron-bridgepackage is absent on r3 → any test using--megatron-to-hf-mode bridgefails withModuleNotFoundError: No module named 'megatron.bridge.models.qwen.qwen_provider'. Tests affected on r3:test_qwen2.5_0.5B_debug_rollout_then_train.pytest_qwen2.5_0.5B_ppo_critic_only_short.pytest_sglang_config_mixed_offload.py/test_sglang_config_mixed_offload_ft.pytest_qwen2.5_0.5B_sglang_config.pysglang is absent on r3 → tests that genuinely require sglang (PD-disagg backends, teacher servers) fail at import / launch:
test_qwen3.6_35B_A3B_pd_mooncake.py— uses--sglang-disaggregation-transfer-backend mooncake; r3 has no sglang to honor ittest_qwen2.5_0.5B_opd_sglang.py— launches an sglang teacher server; exits with code 1PR #55's IPC 15-tuple format mandates the r3 image (cu129 vllm 0.21.0 cannot unpack the new format —
func, args = handleraisesValueError: too many values to unpack (expected 2)). So PR #55 + r3 is a hard pairing. Options:sglang+megatron-bridgeto the r3 image so it becomes a superset of cu129.Open follow-up beyond PR #48
test_qwen3_4B_ckpt.pyMegatron checkpoint merge — orthogonal to IPC, separate PR.test_qwen3_30B_A3B_r3.pyandtest_qwen3_0.6B_parallel_check.py. Needs PR [Feature] Routing replay (R3) for vLLM rollout (vLLM 0.22+, /inference/v1/generate) #49 owner.megatron-bridgemissing on r3 — packaging concern, not slime.