[sammysun0711]

Motivation

Enable MiMo-V2.6-Flash on gfx942 DP Attention (DPA) support with global TP8, DP2 attention (two TP4 groups), EP1 and DFlash speculative decoding based on upstream DP attention support for DFlash

• [Feat] spec: Support CUDA attention DP for DFlash and DFlash2 sgl-project/sglang#46330
• [Refactor] spec: Resolve draft parallel placement from model declarations sgl-project/sglang#46290

Modifications

Based on upstream for DP attention support with DFlash

• Add DP attention support via --enable-dp-attention, --enable-dp-lm-head for attention-TP collectives and graph capture, draft placement, idle-DP handling, and draft KV budgeting/graph controls.
• Support multimodal DP encoding via --mm-enable-dp-encoder (not enabled by default)
• Fix FlyDSL decode dispatch and FlyPA V-tail masking and fix perf regression;
• Add opt-in router fusion SGLANG_MIMO_FUSED_STABLE_ROUTER_WEIGHTS and SWA eviction tuning SGLANG_DFLASH_SWA_TOKEN_INTERVAL, default off.
• Update evaluate guide and script for both static mxfp4->fp8 model prefill and decode with DPA support

Accuracy Tests

Launch server

CHUNKED_PREFILL_SIZE=32768 \
MEM_FRACTION_STATIC=0.75 \
SWA_FULL_TOKENS_RATIO=0.03 \
MAX_RUNNING_REQUESTS=96 \
DISABLE_RADIX_CACHE=1 \
ENABLE_TWO_BATCH_OVERLAP=0 \
SGLANG_FLYPA_MIMO_PREFILL=1 \
SGLANG_AITER_PA_DECODE_IMPL=flydsl \
./launch_mimo_v26_flash_dp2_tp8_ep1_dpa_accuracy_baseline.sh

Run evalscope GSM8K accuracy verification

MODEL_PATH=/models/MiMo-V2.6-Flash-MOPD/ \
MIN_ACCEPT_LENGTH=4.0 \
python3 ./run_evalscope_gsm8k_accuracy.py

┌──────────────────────┬───────────┬────────────┬──────────┬───────┬─────────┐
│ Model                │ Dataset   │ Metric     │ Subset   │   Num │ Score   │
├──────────────────────┼───────────┼────────────┼──────────┼───────┼─────────┤
│ MiMo-V2.6-Flash-MOPD │ GSM8K     │ Accuracy ↑ │ main     │  1319 │ 97.6%   │
└──────────────────────┴───────────┴────────────┴──────────┴───────┴─────────┘ 
"mean_worker_accept_length": 5.56

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[Copilot]

🟡 Changes recommended

Missing test registration and a pytest entry point block CI collection, while the benchmark calls a nonexistent analyzer and a valid DFlash configuration can crash.

5 open issues

• Include the missing benchmark analyzer script · New
• Add an executable pytest entry point · New
• Register the new ROCm tests with the CI suite · New
• Fall back to the resolved token count for DFlash block size · New
• Fix the decode command’s benchmark script path · New

What changed in this PR

Adds MiMo V2.6 Flash data-parallel attention support on gfx942, combining global TP8 with two attention TP4 groups and DFlash speculative decoding.

Changes:

• Isolates draft execution groups, accounts for draft KV memory, and handles idle attention replicas and graph capture.
• Updates FlyDSL decode dispatch and FlyPA tail masking, with optional router fusion and sliding-window eviction tuning.
• Adds regression tests, deployment launchers, and prefill/decode benchmarking guidance.

File	Description
test/​registered/​unit/​spec/​test_dflash_swa_eviction_interval.py	Tests eviction timing and live-page preservation.
test/​registered/​unit/​layers/​moe/​test_mimo_stable_router_weights.py	Tests fused router numerics and graph replay.
test/​registered/​attention/​test_mimo_flypa_nan_tail.py	Tests masking of unused KV tails containing NaNs.
test/​registered/​attention/​test_mimo_flydsl_full_decode.py	Tests FlyDSL decode correctness, warmup, and replay.
test/​registered/​attention/​test_aiter_mimo_vectorized_5d.py	Extends decode dispatch and compilation coverage.
python/​sglang/​srt/​speculative/​draft_parallel.py	Resolves draft placement and isolates execution state.
python/​sglang/​srt/​speculative/​dflash_worker_v2.py	Integrates attention-local drafts, memory budgeting, and idle handling.
python/​sglang/​srt/​speculative/​dflash_info.py	Supports verification padding and deferred metadata initialization.
python/​sglang/​srt/​speculative/​dflash_info_v2.py	Normalizes device identity for reusable buffers.
python/​sglang/​srt/​speculative/​dflash_dp.py	Gathers embeddings when collectives would cross independent drafts.
python/​sglang/​srt/​models/​mimo_vl.py	Exposes vision output dimensions.
python/​sglang/​srt/​models/​mimo_v2.py	Enables data-parallel image and video encoding.
python/​sglang/​srt/​models/​dflash.py	Declares the draft’s single-group execution layout.
python/​sglang/​srt/​model_executor/​runner/​decode_cuda_graph_runner.py	Reformats an existing graph configuration call.
python/​sglang/​srt/​model_executor/​pool_configurator.py	Includes draft KV and fixed allocations in pool sizing.
python/​sglang/​srt/​model_executor/​model_runner.py	Stores draft configuration, layout, and memory budget.
python/​sglang/​srt/​model_executor/​forward_batch_info.py	Initializes empty multimodal positions for idle batches.
python/​sglang/​srt/​mem_cache/​draft_kv_budget.py	Calculates draft KV memory from its attention geometry.
python/​sglang/​srt/​managers/​tp_worker.py	Uses the world-group rank for seed broadcasts.
python/​sglang/​srt/​managers/​scheduler_components/​request_receiver.py	Synchronizes shared-memory consumers within attention groups.
python/​sglang/​srt/​managers/​schedule_batch.py	Adds block-aware DFlash eviction intervals.
python/​sglang/​srt/​layers/​moe/​topk.py	Adds opt-in fused stable router weights.
python/​sglang/​srt/​layers/​moe/​stable_router_weights.py	Implements stable selected-expert weights in Triton.
python/​sglang/​srt/​layers/​attention/​vision.py	Forwards window and sink parameters to attention.
python/​sglang/​srt/​layers/​attention/​flydsl_pa.py	Masks unused V tails and adjusts tile scheduling.
python/​sglang/​srt/​layers/​attention/​aiter_utils.py	Shares FlyDSL decode/verification handling and adds warmup.
python/​sglang/​srt/​layers/​attention/​aiter_backend.py	Prepares ordinary decode alongside speculative verification.
python/​sglang/​srt/​environ.py	Adds eviction and draft graph controls.
python/​sglang/​srt/​distributed/​parallel_state.py	Supports attention-group capture and isolated draft groups.
python/​sglang/​srt/​configs/​draft_parallel.py	Defines draft execution and placement metadata.
python/​sglang/​srt/​arg_groups/​speculative_hook.py	Permits DFlash DP attention with configuration constraints.
evaluation/​run_benchmark_mimo_pro_prefill.sh	Adds benchmark matrices, dataset overrides, and failure reporting.
evaluation/​mimo_v26_flash/​mimo-v2.6-flash-mopd-evaluation-guide.md	Documents DP-attention deployment and benchmark procedures.
evaluation/​mimo_v26_flash/​launch_mimo_v26_flash_dp2_tp8_ep1_dpa_decode_fake_prefill_baseline.sh	Adds a TP8 DP-attention fake-prefill decode launcher.
evaluation/​mimo_v26_flash/​launch_mimo_v26_flash_dp2_tp8_ep1_dpa_accuracy_baseline.sh	Adds a DP-attention accuracy launcher with optional vision DP.
evaluation/​mimo_v26_flash/​launch_mimo_v26_flash_dp2_tp4_ep1_dflash_decode_fake_prefill_baseline.sh	Adds native MXFP4 recipe selection.
evaluation/​benchmark_sglang_prefill.sh	Adds a fixed-concurrency prefill benchmark matrix.
evaluation/​bench_sglang_prefill_sweep.sh	Adds request-rate tuning against a target first-token latency.

_{Review effort: Balanced}