Skip to content

[MRV2] Enable routed-experts capture on r3_offload - #12

Closed
aoshen02 wants to merge 5 commits into
xhx1022:r3_offloadfrom
aoshen02:codex/mrv2-r3-stacked
Closed

aoshen02 wants to merge 5 commits into
xhx1022:r3_offloadfrom
aoshen02:codex/mrv2-r3-stacked

Conversation

@aoshen02

@aoshen02 aoshen02 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • enable routed-experts (R3) capture in Model Runner V2
  • share worker-side capturer/writer lifecycle between Model Runner V1 and V2
  • preserve the existing per-step immutable snapshot used by async scheduling

Why

The R3 implementation on r3_offload only initialized and drained capture
state from Model Runner V1. Model Runner V2 has a split
execute_model()/sample_tokens() lifecycle and returns results through
AsyncOutput, so merely removing the V2 feature guard would run without
publishing valid R3 data.

This change integrates R3 at the Model Runner V2 output boundary. It does not
change the scheduler-visible R3 layout, SHM protocol, KV-sidecar semantics, or
HTTP response format.

Changes

Shared capture lifecycle

  • add RoutedExpertsCaptureState as the common owner of the GPU capturer,
    scheduler-visible writer, full-attention KV group id, and write-task creation
  • move router binding into the capture subsystem so both model runners use the
    same modular and supported monolithic-MoE validation
  • retain the current writer-rank selection and SHM writer implementation from
    r3_offload
  • keep the capturer non-optional for the lifetime of the capture state, and
    close only the optional SHM writer during runner shutdown

Model Runner V2 integration

  • initialize R3 after KV-cache initialization through the existing
    GPUWorker.init_device() hook
  • clear the capture buffer at the beginning of each model step
  • after forward, snapshot the active routing rows and full-attention slot
    mapping into a per-step RoutedExpertsWriteTask
  • carry the task through ExecuteModelState from execute_model() to
    sample_tokens()
  • start the non-blocking D2H copy on the existing AsyncOutput copy stream
  • after the existing copy event completes, write the copied rows to SHM and
    attach routed_experts_slots to ModelRunnerOutput
  • preserve the existing EP-fault query and error path in AsyncOutput

Model Runner V1 refactor

  • use the same RoutedExpertsCaptureState for capturer binding, writer
    ownership, sync publication, and async write-task creation
  • keep the existing sync staging buffers and async immutable snapshot; this is
    a lifecycle refactor, not a data-path semantic change

Configuration

  • remove R3 from Model Runner V2's unsupported-feature list
  • retain the existing fail-closed PP, DCP/PCP, and KV-role validation in
    VllmConfig

Async correctness

The capturer buffer and attention slot mapping are reused by the next step.
The async path therefore keeps the existing clone()-based per-step snapshot
before handing data to the output copy stream. AsyncOutput owns that snapshot
until D2H and SHM publication complete, matching the lifetime model used by
other per-step MRV2 outputs. Snapshot allocation/buffering optimizations are
intentionally left for a separate measured change.

Validation

  • PYTHONPATH=<worktree> /home/aoshen/vllm/.venv/bin/python -m pytest tests/model_executor/test_routed_experts_capture.py -q
    • 22 passed
  • commit hooks passed, including ruff check/format, mypy, SPDX, forbidden
    imports, and configuration validation

Duplicate-work check

No open PR against this fork targets MRV2 enablement on r3_offload.
vllm-project#49184 covers capture from supported monolithic MoE kernels,
not the Model Runner V2 lifecycle and async output integration implemented
here.

AI assistance

AI assistance was used to rebase, resolve conflicts, review the async lifetime,
and prepare tests and documentation. The submitter must review and understand
every changed line before merging.

aoshen02 added 5 commits July 31, 2026 14:27
Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>

Signed-off-by: aoshen02 <aoshen@inferact.ai>
@aoshen02
aoshen02 marked this pull request as ready for review July 31, 2026 14:50
@aoshen02 aoshen02 closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant