[xsxszab]

Overview

MiniMax-M3 enables model-level sequence parallelism (SP) for TP + EP + MegaMoE (#64659), keeping the residual stream sharded by token across TP ranks, but only when pipeline parallelism (PP) is off. This PR keeps SP on under PP, handling stage boundaries the same way as DeepSeek-V4.1: non-last stages all-gather the token shards before returning IntermediateTensors, and the next stage re-shards them on entry. No other open PR enables this combination.

Claims

• MiniMax-M3 with MegaMoE, TP > 1 and PP > 1 now runs the same SP path as PP = 1.
• PP2 and PP4 reproduce PP1's prompt logprobs exactly; GSM8K matches PP1 with and without DSpark.
• Throughput at PP2 × TP2 is unchanged: for each workload, the runs with and without this PR overlap.
• PP = 1 and non-MegaMoE backends are unchanged.

Validation

MiniMaxAI/MiniMax-M3.1-Flash on B300, TP2 + EP + MegaMoE.

Prompt logprobs. Six fixed prompts via /v1/completions with prompt_logprobs: 0 (405 scored positions). PP2 and PP4 with this PR match PP1 at every position (max |Δ| = 0).

GSM8K (1319 questions, chat, thinking disabled):

Config	main	This PR
PP1 × TP2 (SP on)	0.958	unchanged
PP2 × TP2	0.957 (SP off)	0.960
PP1 × TP2 + DSpark (SP on)	0.957	unchanged
PP2 × TP2 + DSpark	0.962 (SP off)	0.951, 0.957

DSpark draft acceptance (num_speculative_tokens=7): 0.524 at PP1; 0.521 and 0.523 at PP2 with this PR; 0.526 at PP2 on main.

Throughput, PP2 × TP2, output tok/s of two interleaved runs per arm:

Workload (in/out, concurrency)	main (SP off)	This PR (SP on)
decode 512/512, c64	2013, 2005	2007, 1994
mixed 2048/256, c64	1654, 1660	1576, 1659
prefill 8192/16, c8	97.6, 98.0	97.8, 100.5

Commands

vllm serve MiniMaxAI/MiniMax-M3.1-Flash --trust-remote-code --max-model-len 16384 \
  --max-num-batched-tokens 16384 --max-num-seqs 64 --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching --enable-expert-parallel --moe-backend deep_gemm_mega_moe \
  --tensor-parallel-size 2 --pipeline-parallel-size 2
# DSpark rows add:
#   --speculative-config '{"method": "dspark", "num_speculative_tokens": 7, "attention_backend": "FLASHINFER_TS", "kv_cache_dtype": "fp8", "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}'

python tests/evals/gsm8k/gsm8k_eval.py --port 8000 --max-concurrency 64 \
  --use-chat-completions --max-tokens 1024 \
  --chat-template-kwargs '{"enable_thinking": false}'

vllm bench serve --model MiniMaxAI/MiniMax-M3.1-Flash --dataset-name random \
  --random-input-len 512 --random-output-len 512 --num-prompts 512 \
  --max-concurrency 64 --ignore-eos --seed 1

curl http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' \
  -d '{"model": "...", "prompt": "...", "max_tokens": 1, "temperature": 0, "prompt_logprobs": 0}'

Details

PP transport sends 1/TP of each IntermediateTensors entry from every TP rank and all-gathers it on the receiver, so stage outputs must cover all tokens and be identical across TP ranks. The change:

• Drop pipeline_parallel_size == 1 from _use_sequence_parallel.
• On entry to a non-first stage, shard the received residual together with hidden_states.
• At a non-last stage exit, pack hidden_states, residual and the aux hidden states into one tensor and all-gather it once. On the last rank, aux states received from earlier stages are already full and are appended after the local gather, as before.
• Build the dense MLP's down_proj with reduce_results=False under SP. With PP > 1 the decoder layer asks each FFN to reduce its own output, but under SP the dense MLP already reduce-scatters the down_proj output, so an extra all-reduce would sum it twice.

AI assistance (Claude) was used to investigate, implement and validate this change. The submitter reviewed every changed line and the validation results.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

✅ @xsxszab, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[xsxszab]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99979 for commit 8d451f598edb.

ℹ️ This PR is 36 commits behind upstream main. Merged upstream main into this branch as 8d451f598edb before starting CI. Pull the branch before pushing more changes.