Conversation
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The R3 implementation on
r3_offloadonly initialized and drained capturestate from Model Runner V1. Model Runner V2 has a split
execute_model()/sample_tokens()lifecycle and returns results throughAsyncOutput, so merely removing the V2 feature guard would run withoutpublishing valid R3 data.
This change integrates R3 at the Model Runner V2 output boundary. It does not
change the scheduler-visible R3 layout, SHM protocol, KV-sidecar semantics, or
HTTP response format.
Changes
Shared capture lifecycle
RoutedExpertsCaptureStateas the common owner of the GPU capturer,scheduler-visible writer, full-attention KV group id, and write-task creation
same modular and supported monolithic-MoE validation
r3_offloadclose only the optional SHM writer during runner shutdown
Model Runner V2 integration
GPUWorker.init_device()hookmapping into a per-step
RoutedExpertsWriteTaskExecuteModelStatefromexecute_model()tosample_tokens()AsyncOutputcopy streamattach
routed_experts_slotstoModelRunnerOutputAsyncOutputModel Runner V1 refactor
RoutedExpertsCaptureStatefor capturer binding, writerownership, sync publication, and async write-task creation
a lifecycle refactor, not a data-path semantic change
Configuration
VllmConfigAsync correctness
The capturer buffer and attention slot mapping are reused by the next step.
The async path therefore keeps the existing
clone()-based per-step snapshotbefore handing data to the output copy stream.
AsyncOutputowns that snapshotuntil D2H and SHM publication complete, matching the lifetime model used by
other per-step MRV2 outputs. Snapshot allocation/buffering optimizations are
intentionally left for a separate measured change.
Validation
PYTHONPATH=<worktree> /home/aoshen/vllm/.venv/bin/python -m pytest tests/model_executor/test_routed_experts_capture.py -q22 passedimports, and configuration validation
Duplicate-work check
No open PR against this fork targets MRV2 enablement on
r3_offload.vllm-project#49184 covers capture from supported monolithic MoE kernels,
not the Model Runner V2 lifecycle and async output integration implemented
here.
AI assistance
AI assistance was used to rebase, resolve conflicts, review the async lifetime,
and prepare tests and documentation. The submitter must review and understand
every changed line before merging.