[xytpai]

Summary

Forward the fixed internal fused_moe -> fused_moe_ call in registered-schema order instead of passing all 38 arguments by keyword.
The public fused_moe API remains keyword-friendly. The registered aiter.fused_moe_ operator, fake implementation, mutation schema, kernels, numerical operations and dispatch policies are unchanged.

The PyTorch dispatcher was repeatedly matching these keyword arguments on every eager MoE invocation. This matters for eager regions within otherwise compiled/captured models.
No eager-body bypass, tensor/pointer cache, compile-key change, or architecture-specific tuning is introduced.

Validation

• 23 passed on gfx1250 for python3 -m pytest -q op_tests/test_moe_forwarding.py against main 89a28cddd4d626212992c0c3cfac1188fb4b10d6 + this patch.
  • All 38 forwarded values and defaults, optional tensors, EP fields and enum conversion.
  • torch.compile(fullgraph=True) still contains aiter.fused_moe_.
  • A4W4/A8W4 × SiLU/SwiGLU/SiTUv2 × M=1/16/257: unchanged reference gate and bitwise output equality against the original keyword dispatcher, including changed-input graph replay.
• 18 passed in a separate old-source/new-source real GPU comparison.
• Existing dispatch regression selection: 15 passed, 103 skipped (architecture-specific cases unavailable on gfx1250).
• GPU-hidden CPU run: 5 passed, 18 skipped. Black/Ruff and git diff --check pass.
• At M256, separate traces for A4W4 and A8W4 have identical kernel names/grid/block sequences.

An initial A8W4 test was incorrectly allowed to select the default FP4 activation. An unchanged-source negative control reproduced the same reference failure and bitwise identical outputs. The final test explicitly selects quant_dtype_a in both arms; no tolerance was relaxed. The initial failure is retained in the experiment logs.

Performance: real MoE, not whole-model throughput

gfx1250, MiniMax-M3 TP4 per-rank operator shape E129, top5, K6144, I768, SwiGLU. Includes real routing, quantization, both expert GEMMs and gather/reduce. Same weights, input and kernels in both arms. This is a single-GPU operator experiment, not a TP4 distributed benchmark.

Independent repeat: 15 shuffled-order paired samples × 160 eager calls; synchronized before each sample and after all calls. Table reports median CPU submission time per call; completion time was recorded separately and closely tracks it. Warmup/compilation/reference/profiling are excluded. AITER_LOG_LEVEL=WARNING in both arms matches the model launch.

Format	M	Original kwargs (µs)	Positional (µs)	Latency reduction
A4W4	1	201.332	194.403	3.44%
A4W4	16	201.809	195.235	3.26%
A4W4	256	202.220	195.594	3.28%
A4W4	2517	261.706	254.607	2.71%
A8W4	1	192.210	185.886	3.29%
A8W4	16	193.766	187.094	3.44%
A8W4	256	193.285	186.724	3.39%
A8W4	2517	253.866	247.165	2.64%

Paired median savings: 6.2–7.1 µs/call in this repeat. These are host-limited eager-call improvements, not faster GPU kernels; graph-only replay need not benefit.

The first independent process also improved all A4W4 shapes but had a noisy/regressing A8W4 M256 sample while emitting per-call untuned-config INFO logs. That run is not discarded or used as a uniform speedup claim; the table above is the independent, model-matched WARNING-level repeat. A8W4 uses the same existing fallback GEMM configuration in both arms.

Full-model profiler-off ABBA on the same ATOM snapshot plus merged #2761 is running separately. No full-model throughput, TTFT or accuracy gain is claimed here. Its synthetic speculative acceptance setting is only appropriate for performance comparisons.

Artifacts on f01: ~/bench/m3_moe_dispatch_20261030/ (logs/gpu_repeat.json, logs/gpu_bench.json, reference/graph logs, and kernel-identity traces).

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7210 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[aiter-bot]

NOT READY FOR HUMAN REVIEW

PR Self-Review

Revision f28808c5e7c8 · mode shadow

Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision.

Questions to answer in the PR description

1. What behavior does this PR change, and why is the existing behavior insufficient?
2. What did you validate, and what remains unverified? Include commands, hardware/configuration and results.
3. What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test?
4. For every changed op_tests/ file, what behavior does it cover and why is it not redundant? Write None if no op test changed.

Edit the PR Self-Review section in the PR description. Include the current revision SHA; this comment refreshes automatically.

Missing or invalid answers

• FAIL — missing ### Reviewed revision in the PR description
• FAIL — missing ### Behavior and intent in the PR description
• FAIL — missing ### Validation and unverified scope in the PR description
• FAIL — missing ### PR-specific question in the PR description
• FAIL — missing ### Changed op tests in the PR description
• FAIL — Reviewed revision must contain at least the first 7 characters of the current head SHA
• FAIL — Behavior and intent has no substantive answer
• FAIL — Validation and unverified scope has no substantive answer
• FAIL — PR-specific question has no substantive answer
• FAIL — changed op test not explained in Changed op tests: op_tests/test_moe_forwarding.py
• FAIL — Changed op tests must explain why each changed op test is necessary and non-redundant

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}

[xytpai]

Follow-up: profiler-off MiniMax-M3 TP4 ABBA

The complete four-process run is now available. Both arms use the same ATOM snapshot + merged #2761, NCCL_PROTO=Simple, TP4, FP8 KV, EAGLE3, and identical recorded prompt hashes. Only the aiter internal MoE call differs.

Performance-only setup: speculative acceptance is forced to 0.5933. Output quality/text is not meaningful in this setup; this is not an accuracy result. This bounded paired workload is not the original 900s aiperf load.

Each process has warmups, 20 short (2517-token) requests, 20 requests with a cached 102784-token prefix + 2517 new tokens, and six measured two-request decode rounds (128 output tokens/request). No profiler. Source imports were verified separately for every process.

Process order	Arm	Short request latency median (ms)	Long-prefix request latency median (ms)	C2 output tokens/s median
1	baseline_a	93.362	109.857	465.79
2	positional_b	93.353	112.449	478.58
3	positional_c	92.680	109.385	477.89
4	baseline_d	89.747	109.738	479.06

Descriptive comparison (mean of the two per-process medians in each arm, not a significance test):

• Short latency: candidate vs baseline +1.60%.
• Long-prefix latency: candidate vs baseline +1.02%.
• C2 throughput: candidate vs baseline +1.23%.

Please judge the raw ABBA spread as well as the aggregate: CPU scheduling/drift can be comparable to a small end-to-end change. The independently repeated operator-level saving in the PR is narrower and does not guarantee a model speedup. Latencies here are end-to-end request times, not exact TTFT or GPU kernel time.

An earlier model launch used invalid host-absolute symlinks inside the container and fell back to image packages; it failed before producing measurements. It was excluded and archived as logs/invalid_model_imports. The table is from the corrected explicit mounts with per-arm import-path assertions.

Artifacts: ~/bench/m3_moe_dispatch_20261030/logs/model_ab/, including all prompt hashes, request timings and generated launch scripts.