[qli88]

Overview

Add Fused heterogeneous MoE support for HY4 preview. If AITER lib is too old to support, it will fall back to default multi-stream MoE. This depends on AITER PR ROCm/aiter#6050.

Claims

HY4 FHMoE Benchmark Results

• Fused: AITER MXFP8 routed and shared experts using FHMoE
• Non-fused: AITER MXFP8 routed experts with a separate shared-expert path
• Tensor parallelism: 8
• CUDA graphs: enabled
• Requests: 2,104 successful, 0 failed

Input 512 / Output 4096

Concurrency	Fused output (tok/s)	Non-fused output (tok/s)	Throughput gain	Fused TPOT (ms)	Non-fused TPOT (ms)	TPOT reduction
1	46.73	44.95	+3.95%	21.367	22.208	3.79%
2	90.47	87.10	+3.87%	22.056	22.902	3.70%
4	144.13	139.90	+3.03%	23.268	23.920	2.73%
8	307.29	303.91	+1.11%	25.950	26.235	1.09%
16	573.97	554.88	+3.44%	27.739	28.667	3.24%
32	970.07	957.65	+1.30%	32.740	33.154	1.25%
64	1593.92	1582.29	+0.73%	39.684	39.955	0.68%
128	2713.30	2664.70	+1.82%	46.474	47.305	1.76%

Input 4096 / Output 512

Concurrency	Fused output (tok/s)	Non-fused output (tok/s)	Throughput gain	Fused TPOT (ms)	Non-fused TPOT (ms)	TPOT reduction
1	45.21	43.57	+3.78%	21.558	22.391	3.72%
2	85.68	82.58	+3.76%	22.458	23.322	3.71%
4	133.19	129.83	+2.59%	23.718	24.293	2.36%
8	264.45	261.44	+1.15%	26.370	26.608	0.90%
16	439.16	424.41	+3.48%	30.136	31.352	3.88%
32	644.07	631.19	+2.04%	39.407	40.134	1.81%
64	860.33	846.61	+1.62%	57.825	58.720	1.52%
128	1099.42	1075.54	+2.22%	90.445	92.307	2.02%

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @qli88.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[AndreasKaratzas]

This PR needs a lot of work. @qli88 you need to carefully read through the changes that you are proposing here. There are a lot of nits and weird spots. Examples include stringent tests that are not really necessary, arbitrary thresholds without justification, gfx logic instead of cdna based logic, and the general feeling that this is not really the most clean and minimal way of introducing this feature.

[AndreasKaratzas]

Aiter ops changes, mxfp8 aiter changed, and more .. this PR should be broken to multiple little PRs

[qli88]

Aiter ops changes, mxfp8 aiter changed, and more .. this PR should be broken to multiple little PRs

@AndreasKaratzas this PR is based on another PR #58509 (approved by Doug and now waiting to be merged) so it looks chunky. I will wait for that PR to get merged and create a new one.