[WIP] Router for vllm by knlnguyen1802 · Pull Request #5 · SamitHuang/slime

knlnguyen1802 · 2026-03-10T03:48:00Z

This is draft router implementation for vllm router

What is solve
It reuse the slime router and just add api compatible for vllm

Issue
Need to verify correctness
Still separate router api and old api of vllm (direct generate via AsyncLLM)

Test plan and debug

bash run-qwen2.5-0.5B-vllm.sh

Debug tip

Can adapt the value of --vllm-gpu-memory-utilization if OOM occur or not enough memory for KVCache error

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

gemini-code-assist · 2026-03-10T03:48:17Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant architectural shift by integrating vLLM as the primary inference backend for the SlimeRouter, replacing the previous SGLang backend. The core innovation lies in a lightweight translation sidecar that seamlessly bridges the API differences between the existing router and the new vLLM engines. This approach ensures that the SlimeRouter and its middleware stack remain entirely unchanged, while enabling the use of vLLM's high-performance inference capabilities. The changes facilitate a more robust and scalable deployment of language models, particularly for reinforcement learning workflows, by providing a flexible and efficient mechanism for managing and interacting with vLLM instances.

Highlights

vLLM Integration with SlimeRouter: Introduced a new architecture to replace the SGLang inference backend with vLLM, leveraging a translation sidecar to maintain compatibility with the existing SlimeRouter without modifications.
Translation Sidecar Implementation: Developed a dedicated FastAPI-based translation sidecar (vllm_translation_sidecar.py) that sits between the SlimeRouter and the vLLM server. This sidecar translates SGLang-format requests to vLLM's OpenAI-compatible API and translates vLLM responses back to SGLang format.
VLLMEngine Enhancements: Updated the VLLMEngine Ray actor to manage the lifecycle of the vLLM server and its co-located translation sidecar, including launching, health checks, and registration with the SlimeRouter.
Flexible Rollout Deployment: Modified the rollout server startup to support both direct vLLM interaction and a router-sidecar mode, allowing for scalable deployment of multiple vLLM engines managed by the SlimeRouter.
API Contract and Abort Strategy: Defined a clear API contract for the translation sidecar, detailing request/response translation, and implemented an abort strategy using vLLM's /pause and /resume endpoints for efficient resource management between generation rounds.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

docs/en/vllm/ROUTER_DESIGN.md
- Added a detailed design document outlining the integration of vLLM with SlimeRouter via a translation sidecar.
run-qwen2.5-0.5B-vllm.sh
- Updated the script to enable the SlimeRouter and include the RadixTreeMiddleware for vLLM rollouts.
slime/backends/vllm_utils/init.py
- Exported TranslationSidecar and run_sidecar for the new vLLM translation sidecar functionality.
slime/backends/vllm_utils/vllm_engine.py
- Modified the VLLMEngine to manage the lifecycle of a vLLM translation sidecar, register with the router, and handle weight versioning.
slime/backends/vllm_utils/vllm_translation_sidecar.py
- Added a new FastAPI application that translates SGLang requests to vLLM API calls and responses, and handles various lifecycle endpoints.
slime/ray/rollout.py
- Updated the vLLM rollout server startup logic to optionally launch a SlimeRouter and multiple vLLM engines with their respective translation sidecars.
slime/rollout/backends/vllm_client.py
- Extended the VLLMClient to support both direct vLLM interaction and a new router-sidecar mode for request generation and abort functionality.
slime/utils/arguments.py
- Added a new command-line argument, --vllm-gpu-memory-utilization, to control vLLM's GPU memory usage.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a significant architectural change by integrating vLLM as a backend via a new router and translation sidecar. However, a medium-severity security issue was identified: the newly introduced translation sidecar has missing authentication and insecure default network binding. This sidecar, which handles sensitive model inference and control requests, binds to all interfaces without access control, potentially exposing the model to unauthorized users. Additionally, the review identified general issues related to potential race conditions, error handling, and documentation/code inconsistencies that require attention.

gemini-code-assist · 2026-03-10T03:51:21Z

+            except Exception:
+                pass


The broad except Exception: pass silently ignores all errors when fetching the weight version from the sidecar. This can lead to using a stale _weight_version, which could cause data consistency issues for caching mechanisms that depend on it. The exception should be logged to help debug potential problems.

Suggested change

except Exception:

pass

except Exception as exc:

logger.warning("Failed to get weight version from sidecar, falling back to local version. Error: %s", exc)

gemini-code-assist · 2026-03-10T03:51:21Z

+            async with self._lock:
+                # We don't actually hold the lock during the request,
+                # just use it to safely add to the tracking set.
+                pass
+
+            resp = await self._client.post(url, json=vllm_payload)
+            self._active_connections.add(resp)


The current implementation for managing _active_connections is not thread-safe and could lead to race conditions. The async with self._lock: pass block is empty, and the modification to _active_connections on line 227 is unprotected. To ensure atomicity in an async context, all operations that modify the shared _active_connections set (including add, discard, and the operations in abort_request) should be wrapped in async with self._lock:. This will prevent inconsistent state when generate and abort_request are called concurrently.

Suggested change

async with self._lock:

# We don't actually hold the lock during the request,

# just use it to safely add to the tracking set.

pass

resp = await self._client.post(url, json=vllm_payload)

self._active_connections.add(resp)

resp = await self._client.post(url, json=vllm_payload)

async with self._lock:

self._active_connections.add(resp)

gemini-code-assist · 2026-03-10T03:51:21Z

+        app.post("/generate")(self.generate)
+        app.get("/health")(self.health)
+        app.get("/health_generate")(self.health_generate)
+        app.post("/abort_request")(self.abort_request)
+        app.get("/flush_cache")(self.flush_cache)
+        app.get("/get_weight_version")(self.get_weight_version)
+        app.post("/set_weight_version")(self.set_weight_version)


The vLLM Translation Sidecar is a new FastAPI application introduced to proxy requests between the SlimeRouter and the vLLM server. By default, the sidecar binds to 0.0.0.0 (all network interfaces) and does not implement any authentication or authorization mechanisms for its endpoints, including /generate, /abort_request, and /set_weight_version. This allows any user with network access to the sidecar's port (default 8100) to perform unauthorized model inference, abort in-flight requests (DoS), or manipulate the model's weight version.

Recommendation: Implement authentication (e.g., API keys or Bearer tokens) for all sidecar endpoints and consider changing the default binding to 127.0.0.1 if the sidecar and router are always co-located.

gemini-code-assist · 2026-03-10T03:51:21Z

+    sidecar_host: str = "0.0.0.0",
+    sidecar_port: int = 8100,


The sidecar application binds to 0.0.0.0 by default, making it accessible from any network interface. In the absence of authentication, this exposes the model and its control endpoints to unauthorized network access.

gemini-code-assist · 2026-03-10T03:51:21Z

+    --tensor-parallel-size <tp_size> \
+    --enable-sleep-mode \
+    --enforce-eager \
+    --gpu-memory-utilization 0.9 \


The example vllm serve command specifies --gpu-memory-utilization 0.9. This value is quite high for a setup where training and inference might be co-located, and it differs from the more conservative default of 0.4 implemented in the code. To prevent potential out-of-memory errors and user confusion, I recommend updating the documentation to reflect a safer default value.

gemini-code-assist · 2026-03-10T03:51:21Z

+import signal
+import sys


The signal and sys modules are imported but not used in this file. These should be removed to maintain code cleanliness.

SamitHuang · 2026-03-16T07:18:22Z

Errors occured:

(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2143565 [0] NCCL INFO Bootstrap: Using ibp24s0:100.126.0.229<0>
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2143565 [0] NCCL INFO cudaDriverVersion 12090
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2143565 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2143565 [0] NCCL INFO Comm config Blocking set to 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO NET/Socket : Using [0]ibp24s0:100.126.0.229<0> [1]ibp41s0f0:100.127.0.67<0> [2]ibp64s0:100.126.0.230<0> [3]i
bp79s0:100.126.0.231<0> [4]ibp94s0:100.126.0.232<0> [5]ibp154s0:100.126.0.233<0> [6]ibp170s0f0:100.127.0.68<0> [7]ibp192s0:100.126.0.234<0> [8]ibp206s0:100.126.0.235<0> [9]ibp220s0:100.
126.0.236<0>
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO Initialized NET plugin Socket
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO Assigned NET plugin Socket to comm
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO Using network Socket
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO ncclCommInitRankConfig comm 0x204474d0 rank 0 nranks 1 cudaDev 0 nvmlDev 4 busId 9d000 commId 0x97d01ebbcf0e
3a61 - Init START
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO Bootstrap timings total 0.000619 (create 0.000032, send 0.000094, recv 0.000252, ring 0.000001, delay 0.0000
01)
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2143565:2144416 [0] NCCL INFO comm 0x204474d0 rank 0 nRanks 1 nNodes 1 localRanks 1 localRank 0 MNNVL 0
...
(MegatronTrainRayActor pid=2143565) sharded_state_dict metadata loaded from the checkpoint: {'distrib_optim_sharding_type': 'fully_sharded_model_space', 'singleton_local_shards': False,
 'chained_optim_avoid_prefix': True}
(MegatronTrainRayActor pid=2143565)  loading release distributed checkpoint from /root/Qwen2.5-0.5B-Instruct_torch_dist/
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:41:47] utils.py:903 - make_tp_sharded_tensor_for_checkpoint received extra kwargs: ['allow_shape_mismatch']
(MegatronTrainRayActor pid=2143565) /root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py:958: FutureWarning: `load_state_dict` is deprecated and will be removed in fut
ure versions. Please use `load` instead.
(MegatronTrainRayActor pid=2143565)   checkpoint.load_state_dict(
(MegatronTrainRayActor pid=2143565) /usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/planner_helpers.py:418: FutureWarning: Please use DTensor instead and we are dep
recating ShardedTensor.
(MegatronTrainRayActor pid=2143565)   device = getattr(value, "device", None)
(MegatronTrainRayActor pid=2143565) /usr/local/lib/python3.12/dist-packages/torch/distributed/checkpoint/default_planner.py:479: FutureWarning: Please use DTensor instead and we are dep
recating ShardedTensor.
(MegatronTrainRayActor pid=2143565)   and md.size != obj.size()
(MegatronTrainRayActor pid=2143565) could not find arguments in the checkpoint ...
(MegatronTrainRayActor pid=2143565)  checkpoint version 3.0
(MegatronTrainRayActor pid=2143565)   successfully loaded checkpoint from /root/Qwen2.5-0.5B-Instruct_torch_dist/ [ t 1/1, p 1/1 ] at iteration 0
(MegatronTrainRayActor pid=2143565) /usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. Y
ou can specify `device_id` in `init_process_group` to mute this warning.
(MegatronTrainRayActor pid=2143565)   warnings.warn(  # warn only once
(MegatronTrainRayActor pid=2143565) sharded_state_dict metadata loaded from the checkpoint: {'distrib_optim_sharding_type': 'fully_sharded_model_space', 'singleton_local_shards': False,
 'chained_optim_avoid_prefix': True}
(MegatronTrainRayActor pid=2143565)  loading release distributed checkpoint from /root/Qwen2.5-0.5B-Instruct_torch_dist/
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:41:49] utils.py:903 - make_tp_sharded_tensor_for_checkpoint received extra kwargs: ['allow_shape_mismatch']
(MegatronTrainRayActor pid=2143565) could not find arguments in the checkpoint ...
(MegatronTrainRayActor pid=2143565)  checkpoint version 3.0
(MegatronTrainRayActor pid=2143565)   successfully loaded checkpoint from /root/Qwen2.5-0.5B-Instruct_torch_dist/ [ t 1/1, p 1/1 ] at iteration 0
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:41:51] timer.py:24 - Timer train_wait start
(VLLMEngine pid=2144344) INFO:     Started server process [2145677]
(VLLMEngine pid=2144344) INFO:     Waiting for application startup.
(VLLMEngine pid=2144344) INFO:     Application startup complete.
(VLLMEngine pid=2144344) INFO:     Uvicorn running on http://0.0.0.0:15100 (Press CTRL+C to quit)
(RolloutManager pid=2143282) INFO:     10.248.12.102:54122 - "POST /add_worker?url=http%3A%2F%2F10.248.12.102%3A15100 HTTP/1.1" 200 OK
(RolloutManager pid=2143282) [2026-03-15 17:42:18] data_source.py:147 - Checkpoint /root/Qwen2.5-0.5B-Instruct_torch_dist/rollout/global_dataset_state_dict_-1.pt does not exist.
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:42:18] timer.py:24 - Timer update_weights start
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:42:18] update_weight_from_distributed.py:514 - vLLM weight transfer via NcclBridge: addr=10.248.12.102 port=57879 world_size=2 device=
0 CVD=4
(VLLMEngine pid=2144344) INFO:     10.248.12.102:34242 - "GET /health HTTP/1.1" 200 OK
(MegatronTrainRayActor pid=2143565) INFO 03-15 17:42:34 [pynccl.py:111] vLLM is using nccl==2.27.5
Traceback (most recent call last):
  File "/root/slime/train.py", line 100, in <module>
    train(args)
  File "/root/slime/train.py", line 26, in train
    actor_model.update_weights()
  File "/root/slime/slime/ray/actor_group.py", line 134, in update_weights
    return ray.get([actor.update_weights.remote() for actor in self._actor_handlers])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2981, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1012, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::MegatronTrainRayActor.update_weights() (pid=2143565, ip=10.248.12.102, actor_id=07e53044bed1e4d87bb4b7f802000000, repr=<slime.backends.me
gatron_utils.actor.MegatronTrainRayActor object at 0x7ffe80266ed0>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/timer.py", line 78, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 550, in update_weights
    self.weight_updater.connect_rollout_engines(
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 223, in connect_rollout_engines
    self._model_update_groups = connect_rollout_engines_from_distributed(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 520, in connect_rollout_engines_from_distributed
    model_update_groups = _NcclBridge(
                          ^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 132, in __init__
    raise RuntimeError(f"NcclBridge init failed: {msg}")
RuntimeError: NcclBridge init failed: error: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
Traceback (most recent call last):
  File "/root/slime/train.py", line 100, in <module>
    train(args)
  File "/root/slime/train.py", line 26, in train
    actor_model.update_weights()
  File "/root/slime/slime/ray/actor_group.py", line 134, in update_weights
    return ray.get([actor.update_weights.remote() for actor in self._actor_handlers])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2981, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Bootstrap: Using ibp24s0:100.126.0.229<0>^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1012, in get_objects

    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::MegatronTrainRayActor.update_weights() (pid=2143565, ip=10.248.12.102, actor_id=07e53044bed1e4d87bb4b7f802000000, repr=<slime.backends.me
gatron_utils.actor.MegatronTrainRayActor object at 0x7ffe80266ed0>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/utils/timer.py", line 78, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/actor.py", line 550, in update_weights
    self.weight_updater.connect_rollout_engines(
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 223, in connect_rollout_engines
    self._model_update_groups = connect_rollout_engines_from_distributed(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 520, in connect_rollout_engines_from_distributed
    model_update_groups = _NcclBridge(
                          ^^^^^^^^^^^^
  File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 132, in __init__
    raise RuntimeError(f"NcclBridge init failed: {msg}")
RuntimeError: NcclBridge init failed: error: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
RuntimeError: NcclBridge init failed: error: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO cudaDriverVersion 12090
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NET/Socket : Using [0]ibp24s0:100.126.0.229<0> [1]ibp41s0f0:100.127.0.67<0> [2]ibp64s0:100.126.0.230<0> [3]i
bp79s0:100.126.0.231<0> [4]ibp94s0:100.126.0.232<0> [5]ibp154s0:100.126.0.233<0> [6]ibp170s0f0:100.127.0.68<0> [7]ibp192s0:100.126.0.234<0> [8]ibp206s0:100.126.0.235<0> [9]ibp220s0:100.
126.0.236<0>
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Initialized NET plugin Socket
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Assigned NET plugin Socket to comm
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Using network Socket
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO ncclCommInitRank comm 0x14e796d0 rank 0 nranks 2 cudaDev 0 nvmlDev 4 busId 9d000 commId 0x7035e6c80dc9a144 -
 Init START
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Bootstrap timings total 0.000660 (create 0.000026, send 0.000074, recv 0.000278, ring 0.000031, delay 0.0000
01)
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO comm 0x14e796d0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 00/04 : 0 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 01/04 : 0 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 02/04 : 0 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 03/04 : 0 1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO P2P Chunksize set to 131072
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2146022 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 124
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2146020 [0] NCCL INFO [Proxy Service] Device 0 CPU core 167
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 00 : 0[4] -> 1[5] via SHM/direct/direct
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 01 : 0[4] -> 1[5] via SHM/direct/direct
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 02 : 0[4] -> 1[5] via SHM/direct/direct
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO Channel 03 : 0[4] -> 1[5] via SHM/direct/direct
(MegatronTrainRayActor pid=2143565)
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:42:35] hk01dgx028:2145689:2145689 [0] misc/shmutils.cc:93 NCCL WARN Call to open failed: No such file or directory
(MegatronTrainRayActor pid=2143565)
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:42:35] hk01dgx028:2145689:2145689 [0] misc/shmutils.cc:129 NCCL WARN Error while attaching to shared memory segment /dev/shm/nccl-�� (
size 0), error: No such file or directory (2)
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO transport/shm.cc:634 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO transport/shm.cc:169 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO transport.cc:198 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO transport/generic.cc:19 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO init.cc:1161 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO init.cc:1449 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO init.cc:1832 -> 2
(MegatronTrainRayActor pid=2143565) hk01dgx028:2145689:2145689 [0] NCCL INFO init.cc:1858 -> 2
(MegatronTrainRayActor pid=2143565) Traceback (most recent call last):
(MegatronTrainRayActor pid=2143565) [2026-03-15 17:42:35] timer.py:32 - Timer update_weights end (elapsed: 16.6s)
(MegatronTrainRayActor pid=2143565)   File "/root/slime/slime/backends/megatron_utils/update_weight/update_weight_from_distributed.py", line 68, in _nccl_bridge_worker
(MegatronTrainRayActor pid=2143565)     comm = PyNcclCommunicator(pg, device=device)
(MegatronTrainRayActor pid=2143565)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(MegatronTrainRayActor pid=2143565)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 139, in __init__
(MegatronTrainRayActor pid=2143565)     self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(MegatronTrainRayActor pid=2143565)                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(MegatronTrainRayActor pid=2143565)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 407, in ncclCommInitRank
(MegatronTrainRayActor pid=2143565)     self.NCCL_CHECK(
(MegatronTrainRayActor pid=2143565)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 373, in NCCL_CHECK
(MegatronTrainRayActor pid=2143565)     raise RuntimeError(f"NCCL error: {error_str}")
(MegatronTrainRayActor pid=2143565) RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
[2026-03-15 17:42:35] selector_events.py:1068 - socket.send() raised exception.
[2026-03-15 17:42:35] selector_events.py:1068 - socket.send() raised exception.
[2026-03-15 17:42:35] selector_events.py:1068 - socket.send() raised exception.
[2026-03-15 17:42:35] selector_events.py:1068 - socket.send() raised exception.

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

SamitHuang · 2026-03-17T02:15:48Z

(RolloutManager pid=4095972) [2026-03-16 19:17:17] rollout.py:1252 - perf 495: {'rollout/response_len/mean': 0.0, 'rollout/response_len/median': 0.0, 'rollout/response_len/max': 0, 'rol
lout/response_len/min': 0, 'rollout/zero_std/count_0': 8, 'rollout/zero_std/count_1': 1, 'rollout/repetition_frac': 0.0, 'rollout/truncated_ratio': 0.1015625, 'perf/rollout_time': 14.90
4117584228516, 'perf/tokens_per_gpu_per_sec': 0.0, 'perf/longest_sample_tokens_per_sec': 0.0, 'perf/effective_tokens_per_gpu_per_sec': 0.0, 'perf/longest_effective_sample_tokens_per_sec
': 0.0}

knlnguyen1802 · 2026-03-17T02:22:16Z

(RolloutManager pid=4095972) [2026-03-16 19:17:17] rollout.py:1252 - perf 495: {'rollout/response_len/mean': 0.0, 'rollout/response_len/median': 0.0, 'rollout/response_len/max': 0, 'rol lout/response_len/min': 0, 'rollout/zero_std/count_0': 8, 'rollout/zero_std/count_1': 1, 'rollout/repetition_frac': 0.0, 'rollout/truncated_ratio': 0.1015625, 'perf/rollout_time': 14.90 4117584228516, 'perf/tokens_per_gpu_per_sec': 0.0, 'perf/longest_sample_tokens_per_sec': 0.0, 'perf/effective_tokens_per_gpu_per_sec': 0.0, 'perf/longest_effective_sample_tokens_per_sec ': 0.0}

let's me check it. It look like rollout return nothing

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

SamitHuang · 2026-03-18T02:35:55Z

yellow - this PR run

looks normal, but the loss curve is a bit spiking.

* temp save rfc Signed-off-by: SamitHuang <285365963@qq.com> * add plan Signed-off-by: SamitHuang <285365963@qq.com> * update Signed-off-by: SamitHuang <285365963@qq.com> * [docker] remove true on policy patches (THUDM#1661) Co-authored-by: Copilot <copilot@github.com> * [fix]: Qwen3.5-35B-A3B 8-GPU: set TP size to 2 for num_query_groups=2 (THUDM#1662) * Remove FSDP support (THUDM#1664) Co-authored-by: Copilot <copilot@github.com> * docs: add OpenClaw-RL to projects built upon slime (THUDM#1635) * qwen2.5 0.5b non-colocate (first attempt ok, but nccl error later) Signed-off-by: samithuang <285365963@qq.com> * add convert script * add setup doc * Support setting update weights in sglang_config (THUDM#1665) Co-authored-by: Copilot <copilot@github.com> * fix nccl error by NcclBridge subprocess * eliminate gpu to cpu weight transfer Signed-off-by: samithuang <285365963@qq.com> * Revise weight synchronization strategy in goal plan Reorder weight synchronization support for colocate and non-colocate scenarios in the goal plan. * [fix] Fix numerical accuracy issue in dynamic sampling filter (THUDM#1674) * sync from internal (THUDM#1677) Co-authored-by: Copilot <copilot@github.com> * bugfixes from community (THUDM#1678) Co-authored-by: Copilot <copilot@github.com> Co-authored-by: yueming-yuan <yym022502@gmail.com> Co-authored-by: coding-famer <chenhegu0109@gmail.com> * Fix: pass return_tensors in text_kwargs for transformers>=5.0.0 compatibility (THUDM#1648) * Fix missing packed_seq_params in bshd qkv_format (THUDM#1649) * [Multimodal][Model] Qwen3.5 VL training example/support (THUDM#1676) * update docs (THUDM#1680) Co-authored-by: Copilot <copilot@github.com> * update docs (THUDM#1681) Co-authored-by: Copilot <copilot@github.com> * support offloading non-updatable server (THUDM#1668) Co-authored-by: Copilot <copilot@github.com> * bugfix (THUDM#1685) Co-authored-by: Copilot <copilot@github.com> * fix: handle Qwen3.5 in quantize_params_fp8 (THUDM#1683) * bugfix (THUDM#1687) Co-authored-by: Copilot <copilot@github.com> * Fix Qwen3.5 & Qwen3-Next linear attention cu_seqlens missing (THUDM#1686) Co-authored-by: benyi <huangliangmeng.hlm@alibaba-inc.com> * fix: use semantic version comparison for PyTorch >= 2.6 detection (THUDM#1667) * [Fix] Minor fix for properly finishing / flushing wandb logging metrics at exit (THUDM#1592) Co-authored-by: Zilin Zhu <zhuzilinallen@gmail.com> * Autofix/issue 1578 hf2megatron arg suffix (THUDM#1636) * bugfix (THUDM#1688) Co-authored-by: Copilot <copilot@github.com> * fix(examples): update strands_sglang example to v0.3.x API (THUDM#1684) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * [docker] cherry pick qwen3.5 bugfix (THUDM#1691) Co-authored-by: Copilot <copilot@github.com> * bugfix/fix Qwen3.5 dense model precision bug in TP_SIZE>1 from sglang (THUDM#1705) * Fix/qwen3 5 mtp bridge (THUDM#1702) Co-authored-by: benyi <huangliangmeng.hlm@alibaba-inc.com> * support epd for glm4.6v (THUDM#1704) * [docker] support epd for glm4.6v (THUDM#1707) Co-authored-by: Copilot <copilot@github.com> * remove script * [docker] store v0.5.9 patch (THUDM#1710) Co-authored-by: Copilot <copilot@github.com> * Add GLM-4.7-Flash MTP training support (THUDM#1712) * [release] bump to v0.2.3 (THUDM#1682) Co-authored-by: Copilot <copilot@github.com> * feat: add GLM-4.6V MoE VL bridge with CP support (THUDM#1715) Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve rope_theta from rope_parameters dict in HF config validation (THUDM#1720) Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [docker] patches for glm4.6v, kimi k2.5 and dsa cp only (THUDM#1722) Co-authored-by: Copilot <copilot@github.com> * [docker] support IndexCache * Fix CUDA IPC cache leaks during weight updates (THUDM#1731) Co-authored-by: Copilot <copilot@github.com> * [docker] update megatron (THUDM#1729) Co-authored-by: Copilot <copilot@github.com> * [docker] Fix IndexCache with mla model (THUDM#1736) Co-authored-by: Copilot <copilot@github.com> * [slime-router] support pd disaggregation and remove radix tree middleware (THUDM#1735) * Fix glm4v megatron bridge (THUDM#1738) Co-authored-by: Copilot <copilot@github.com> * [docker] update sglang patch (THUDM#1743) Co-authored-by: Copilot <copilot@github.com> * feat: GLM4V multimodal support improvements (THUDM#1745) Co-authored-by: Copilot <copilot@github.com> * feat: placeholder worker type, metrics router, and GPQA letter range (THUDM#1746) Co-authored-by: Copilot <copilot@github.com> * always enable_metrics and remove dp context (THUDM#1747) Co-authored-by: Copilot <copilot@github.com> * fix: resolve SP/CP gradient inflation in FLA (linear attention) layers (THUDM#1748) Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update MTP example configs, rename GLM-4.5 to GLM-4.7, clean scripts (THUDM#1749) Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Support qwen3.5 loss mask for multi-turn SFT (THUDM#1742) Co-authored-by: benyi <huangliangmeng.hlm@alibaba-inc.com> * fix: propagate moe_token_dispatcher_type in bridge model provider (THUDM#1737) * fix: resolve rope_theta from rope_parameters in DeepseekV32Bridge (THUDM#1734) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * chore: translate remaining Chinese comments to English (THUDM#1726) * feat: add Qwen3.5-4B model support (THUDM#1721) * fix: http_utils. disable system proxy for internal SGLang httpx clients (THUDM#1714) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: auto-detect GPUs in qwen3-4b script (THUDM#1700) * fix: quote `$MOE_LAYER_FREQ` (THUDM#1689) * disable router health_check and allow prompt_data is None (THUDM#1751) Co-authored-by: Copilot <copilot@github.com> * Router for vllm (#5) * Draft router design Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add vllm router Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add router to script Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix gpu memory utilization Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix output token ids Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add more nccl flag Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> --------- Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * small fix on qwen3-235b-a22b launch script (THUDM#1719) * sync internal bugfix (THUDM#1765) Co-authored-by: Copilot <copilot@github.com> * Fix uploading sglang metrics to wandb (THUDM#1768) Co-authored-by: Copilot <copilot@github.com> * use zhuzilin/sgl-router for sglang-router (THUDM#1770) Co-authored-by: Copilot <copilot@github.com> * [docker] update sgl-router (THUDM#1772) Co-authored-by: Copilot <copilot@github.com> * [Multimodal] Add Multimodal OPD support (THUDM#1760) * refactor: remove slime router (THUDM#1773) Co-authored-by: Copilot <copilot@github.com> * Add rollout trace timeline viewer (THUDM#1776) Co-authored-by: Hanyu Zhang <hanyu.zhang@aminer.cn> * [Fix] Fix duplicate Megatron LR scheduler resume when optimizer state is not loaded (THUDM#1775) * Support FP8 conversion for Qwen3.5 (THUDM#1769) * fix typo (THUDM#1759) Co-authored-by: shiqirui <shiqirui@kupasai.com> * [Fix]Fix some bugs/clean up (THUDM#1756) * (fix):not have encoder_only attr cause run failed (THUDM#1741) Co-authored-by: wangch <wangch@wangchdeMacBook-Air.local> * update docs * remove redundant envvar * some minor cleanup * [release] bump to v0.2.4 (THUDM#1777) Co-authored-by: Copilot <copilot@github.com> * Plan refactor vllm/sglang Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Code implemented Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix port Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug MOE weight sync * Fix bug vllm transfer weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix weight sync Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Change name config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * pass critic role through to create RayTrainGroup (THUDM#1797) * fix qwen3.5 397B converting error when enable expert parallel (THUDM#1799) Co-authored-by: 周鹤云 <zhouheyun@xiaohongshu.com> * fix(geo3k-vlm-sft): remove --apply-chat-template from SFT launch script (THUDM#1791) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add host memory metrics to available_memory function (THUDM#1764) * [WIP] fix loss oom (THUDM#1788) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * sync from internal (THUDM#1805) * sync from internal (THUDM#1807) * feat: add npu patch for qwen3-vl-8b grpo & ppo (#1750) Signed-off-by: cjy0x <isjunyi.chen@gmail.com> Co-authored-by: shiyuan680 <917935075@qq.com> Co-authored-by: PengchengShi00 <spc117369@gmail.com> * fix missing position_ids in log-prob forward step (THUDM#1809) * feat: add support for including missing weights from origin HF checkp… (THUDM#1812) * [Fix] Initialize grad_norm before found_inf skip path (THUDM#1762) * [conda] Add install custom sgl-router to build_conda.sh (THUDM#1813) * Revert no_grad for entropy to prevent comm stuck in dsa (THUDM#1822) * Add fallback for get_seqlen_balanced_partitions (THUDM#1823) * Resolve review Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Try colocated vllm weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * docs: add Relax to notable projects in README (THUDM#1834) * Bugfix: use cpu instead of cuda in convert_torch_dist_to_hf.py when --add-missing-from-origin-hf is set (THUDM#1828) * [fix] eval sample logging when sample is a list (THUDM#1836) * [Draft] Local runable dev Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * [Fix] Fix cuda-python pin in build_conda.sh (THUDM#1827) * fix entropy bug and update code (THUDM#1846) * Revert "Add fallback for get_seqlen_balanced_partitions" (THUDM#1848) * fix (THUDM#1849) * Fix offload train Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add support for NVIDIA DGX Spark (GB10 / sm_121a, arm64) (THUDM#1835) * Fix offload train Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload_rollout Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix vllm offload Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload traing Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * refactor/ppo (THUDM#1856) * [docker] cleanup sglang patch (THUDM#1859) * [docker] update v0.5.9 patch * Rename critic config to megatron config (THUDM#1866) * [Fix] Use Ray ObjectRef await instead of asyncio.to_thread in distributed POST (THUDM#1873) * chore: include length context in slice_log_prob_with_cp assert (THUDM#1862) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [docker] upgrade megatron to 1dcf0dafa (THUDM#1867) * fix ppo value head load bugs (THUDM#1878) * [docker] upgrade sglang to v0.5.10.post1 (THUDM#1874) * [docs] update docs * [docker] update megatron-bridge and add qwen3.6 tests (THUDM#1884) * fix lint * Fix(checkpoint): add resume/pause in save_model() for offload_train (fixes THUDM#1886) (THUDM#1888) * fix ppo value offload bugs (THUDM#1882) * fix qwen3.6 hf config validation bug (THUDM#1889) * Add missing metrics to log (THUDM#1890) * fix(qwen3_next): use torch.get_default_dtype() — get_current_dtype do… (THUDM#1883) Co-authored-by: yeqinghe <yeqinghe@MacBook-Pro-6.local> * Fix location error in install script (THUDM#1877) * Only allow --allgather-cp for DSA model (THUDM#1891) * Migrate internal feature (THUDM#1897) * [Fix] Fix distributed POST actor concurrency split (THUDM#1880) Co-authored-by: Zilin Zhu <zhuzilinallen@gmail.com> * Fix CI: update rollout_data_postprocess plugin contract for new call site (THUDM#1902) Co-authored-by: jingshenghang <shenghang.jing@aminer.cn> * Patch Megatron TP grad coalesce to chunked all-reduce (THUDM#1899) * fix: harden retool rollout against multi-turn / retry desync (THUDM#1861) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix log file Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix import engine group Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix rebase code Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> --------- Signed-off-by: SamitHuang <285365963@qq.com> Signed-off-by: samithuang <285365963@qq.com> Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Signed-off-by: cjy0x <isjunyi.chen@gmail.com> Co-authored-by: SamitHuang <285365963@qq.com> Co-authored-by: Zilin Zhu <zhuzilinallen@gmail.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: none0663 <none0663@outlook.com> Co-authored-by: Yinjie Wang <yinjie@uchicago.edu> Co-authored-by: Fengqing Jiang <43953876+Django-Jiang@users.noreply.github.com> Co-authored-by: yueming-yuan <yym022502@gmail.com> Co-authored-by: coding-famer <chenhegu0109@gmail.com> Co-authored-by: Lawrence Wu <lawrence.wu@harmonic.fun> Co-authored-by: huang3eng <huang3eng@gmail.com> Co-authored-by: benyi <huangliangmeng.hlm@alibaba-inc.com> Co-authored-by: Aaron Batilo <AaronBatilo@gmail.com> Co-authored-by: Silun Wang <igeekwang@gmail.com> Co-authored-by: Chengxing Xie <91449279+yitianlian@users.noreply.github.com> Co-authored-by: Yuan He <33579950+Lawhy@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Mor Zusman <mor.zusmann@gmail.com> Co-authored-by: append-only <shw20010329@163.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Nan Jiang <59716405+nanjiangwill@users.noreply.github.com> Co-authored-by: Xuan Wang <49010704+stevewx@users.noreply.github.com> Co-authored-by: Hubert Wang <huberthyw@gmail.com> Co-authored-by: Hou Shihao <shhou007@gmail.com> Co-authored-by: DongzhuoranZhou <110855293+DongzhuoranZhou@users.noreply.github.com> Co-authored-by: Ailuntz <130897222+ailuntz@users.noreply.github.com> Co-authored-by: Zhuohao Li <garrick0508@gmail.com> Co-authored-by: Hanyu Zhang <hanyu.zhang@aminer.cn> Co-authored-by: Kang Yu <kangy.me@gmail.com> Co-authored-by: peterjc123 <peter_jiachen@163.com> Co-authored-by: qrskannbara <94727257+albaNnaksqr@users.noreply.github.com> Co-authored-by: shiqirui <shiqirui@kupasai.com> Co-authored-by: wangyufak <wangch9@xiaopeng.com> Co-authored-by: wangch <wangch@wangchdeMacBook-Air.local> Co-authored-by: Xintong Li <znculee@gmail.com> Co-authored-by: TM <tianmingxu.tmxu@gmail.com> Co-authored-by: 周鹤云 <zhouheyun@xiaohongshu.com> Co-authored-by: LiLei <77353389+lilei199908@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: cjy0x <isjunyi.chen@gmail.com> Co-authored-by: shiyuan680 <917935075@qq.com> Co-authored-by: PengchengShi00 <spc117369@gmail.com> Co-authored-by: 杨睿 <595403043@qq.com> Co-authored-by: Mathew Han <49226490+mathewjhan@users.noreply.github.com> Co-authored-by: haoxuanJIA <116806014+boots-coder@users.noreply.github.com> Co-authored-by: ryang <38470282+ryang-max@users.noreply.github.com> Co-authored-by: Leo Fan <84952531+leofan-lab@users.noreply.github.com> Co-authored-by: Long Yijun <156500868+Procrastinatorrrr@users.noreply.github.com> Co-authored-by: HeatherLiuzh <heather996lzh@gmail.com> Co-authored-by: yeqinghe <yeqinghe@MacBook-Pro-6.local> Co-authored-by: tao W <122036357+selfanti@users.noreply.github.com> Co-authored-by: jingshenghang <48083555+jingshenghang@users.noreply.github.com> Co-authored-by: jingshenghang <shenghang.jing@aminer.cn>

* temp save rfc Signed-off-by: SamitHuang <285365963@qq.com> * add plan Signed-off-by: SamitHuang <285365963@qq.com> * update Signed-off-by: SamitHuang <285365963@qq.com> * qwen2.5 0.5b non-colocate (first attempt ok, but nccl error later) Signed-off-by: samithuang <285365963@qq.com> * add convert script * add setup doc * fix nccl error by NcclBridge subprocess * eliminate gpu to cpu weight transfer Signed-off-by: samithuang <285365963@qq.com> * Revise weight synchronization strategy in goal plan Reorder weight synchronization support for colocate and non-colocate scenarios in the goal plan. * Router for vllm (#5) * Draft router design Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add vllm router Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add router to script Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix gpu memory utilization Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix output token ids Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Add more nccl flag Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> --------- Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Plan refactor vllm/sglang Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Code implemented Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix port Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix bug MOE weight sync * Fix bug vllm transfer weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix weight sync Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Change name config Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Resolve review Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Try colocated vllm weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * [Draft] Local runable dev Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload train Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload train Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload_rollout Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix vllm offload Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload traing Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix offload weight Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix CI: update rollout_data_postprocess plugin contract for new call site (THUDM#1902) Co-authored-by: jingshenghang <shenghang.jing@aminer.cn> * Patch Megatron TP grad coalesce to chunked all-reduce (THUDM#1899) * fix: harden retool rollout against multi-turn / retry desync (THUDM#1861) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix log file Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix import engine group Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> * Fix rebase code Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> --------- Signed-off-by: SamitHuang <285365963@qq.com> Signed-off-by: samithuang <285365963@qq.com> Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Co-authored-by: SamitHuang <285365963@qq.com> Co-authored-by: jingshenghang <48083555+jingshenghang@users.noreply.github.com> Co-authored-by: jingshenghang <shenghang.jing@aminer.cn> Co-authored-by: Leo Fan <84952531+leofan-lab@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

knlnguyen1802 added 5 commits March 6, 2026 11:09

Draft router design

be91022

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

Add vllm router

d644f19

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

Add router to script

408c0bd

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

Fix gpu memory utilization

dce676a

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

Fix output token ids

843b508

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

gemini-code-assist Bot reviewed Mar 10, 2026

View reviewed changes

SamitHuang mentioned this pull request Mar 12, 2026

[Don't Merge] Roadmap and Dev Plan #3

Draft

10 tasks

Add more nccl flag

0a1c053

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

Fix bug

6dc2916

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>

SamitHuang approved these changes Mar 23, 2026

View reviewed changes

SamitHuang merged commit d480da0 into SamitHuang:dev_vllm Mar 23, 2026

Navigation Menu

[WIP] Router for vllm - #5

[WIP] Router for vllm#5
SamitHuang merged 7 commits into
SamitHuang:dev_vllmfrom
knlnguyen1802:router_local_dev

knlnguyen1802 commented Mar 10, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Uh oh!

SamitHuang commented Mar 16, 2026

Uh oh!

SamitHuang commented Mar 17, 2026

Uh oh!

knlnguyen1802 commented Mar 17, 2026

Uh oh!

SamitHuang commented Mar 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-            except Exception:
-                pass
+            except Exception as exc:
+                logger.warning("Failed to get weight version from sidecar, falling back to local version. Error: %s", exc)

Conversation

knlnguyen1802 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan and debug

Debug tip

Uh oh!

gemini-code-assist Bot commented Mar 10, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

SamitHuang commented Mar 16, 2026

Uh oh!

SamitHuang commented Mar 17, 2026

Uh oh!

knlnguyen1802 commented Mar 17, 2026

Uh oh!

SamitHuang commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

knlnguyen1802 commented Mar 10, 2026 •

edited

Loading

SamitHuang commented Mar 18, 2026 •

edited

Loading