Skip to content

fix(moe): restore TP MoE num_local_tokens - #2905

Merged
valarLip merged 6 commits into
ROCm:mainfrom
JiaoliangYu:fix/restore-tp-token-count
Nov 3, 2026
Merged

valarLip merged 6 commits into
ROCm:mainfrom
JiaoliangYu:fix/restore-tp-token-count

Conversation

@JiaoliangYu

@JiaoliangYu JiaoliangYu commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

#2776 discarded scheduled_tokens_gpu whenever use_ep was false, so pure TP lost the real-token count introduced by #2725. Restore the count for ungathered TP rows; the existing MoE methods forward it to AITER as num_local_tokens. DP/PCP-gathered rows still receive None, and pass_scheduled_tokens_gpu rejects row subsets that do not match the current pass.

Run the routed-row regression tests on CPU under AITER/Triton stubs. During this test module's import, bypass CUDA custom-op registration so fake enum defaults do not enter schema inference or leave stub kernels in PyTorch's process-wide registry. The TP/EP regression exercises the production _run_experts and pass_scheduled_tokens_gpu together: a complete local pass receives its count, while gathered rows and row subsets do not.

Dependency and backend scope: ROCm/aiter#7294 is still unmerged. Restoring num_local_tokens on older AITER with padded CUDA-graph shapes can reintroduce its atomic GEMM2 padding-sentinel regression. The stage2 paths covered by that dependency are FlyDSL, FlyDSL-v2, and native-MXFP4. The current target configuration does not use Opus GEMM2. Keep this PR in draft while #7294 is unmerged and the requested pure-TP decode performance comparison remains pending.

Validation:

  • Python 3.12.15 with CPU PyTorch 2.15.0, without AITER or Triton: tests/test_pass_scheduled_tokens.py and tests/model_ops/test_dp_gather_routed_rows.py completed with 10 passed and no skips.
  • Full non-GPU suite in a native Linux checkout, using the same exclusions as Pre Checkin: 10,698 passed, 1,139 skipped, 3 xfailed, with no failures.
  • Pre Checkin on 9705a5e76 passed all checks. The CPU job reported 11,007 passed, 1,154 skipped, 3 xfailed and executed all seven routed-row tests successfully. The offline inference smoke test also passed.
  • Black (--target-version py312), Ruff, and git diff --check passed for the changed file.
  • GSM8K was not run. Pure-TP decode performance validation remains pending.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2905 --add-label <label>

@JiaoliangYu
JiaoliangYu marked this pull request as ready for review November 3, 2026 03:06

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

pure-TP decode perf with aiter#7294 vs main, and gsm8k for a padded cudagraph bucket?

ci green.

Comment thread atom/model_ops/moe.py Resolved
Comment thread atom/model_ops/moe.py Resolved
@JiaoliangYu
JiaoliangYu marked this pull request as draft November 3, 2026 10:06
@JiaoliangYu
JiaoliangYu marked this pull request as ready for review November 3, 2026 13:09

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

pure-TP decode perf with aiter#7294 vs main, and gsm8k for a padded cudagraph bucket on the pure-TP path?

ci green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants