Skip to content

Remove FlyDSL fallback machinery from MoE tuning (FlyDSL now mandatory) - #4025

Merged
coderfeli merged 1 commit into
mainfrom
clean_moe_flydsl_fallback
Jul 8, 2026
Merged

Remove FlyDSL fallback machinery from MoE tuning (FlyDSL now mandatory)#4025
coderfeli merged 1 commit into
mainfrom
clean_moe_flydsl_fallback

Conversation

@coderfeli

Copy link
Copy Markdown
Collaborator

Summary

FlyDSL is now a required dependency for MoE tuning, so the FlyDSL→CK fallback machinery is dead weight. Previously the tuner recorded, alongside every winning flydsl_ config, a companion CK config tagged flydsl_fallback, and the runtime swapped it in when is_flydsl_available() was False. This PR removes that generation + consumption path and the associated _tag data.

Changes

Tuner — csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py (stops generating fallbacks)

  • Removed FLYDSL_FALLBACK_TAG, the self._flydsl_fallbacks accumulator, the _non_flydsl_best computation, and the best_has_flydsl block that merged a CK stage1+stage2 fallback row.
  • result_to_csv no longer writes a _tag column or appends fallback rows. It now just drops any legacy _tag column/rows read from older tuned files, then writes a plain deduped table.

Runtime — aiter/fused_moe.py (stops consuming fallbacks)

  • Removed _flydsl_fallback_cache, get_flydsl_fallback_cfgs(), and the runtime block that substituted a fallback cfg (or dropped to default heuristics) when FlyDSL was unavailable.
  • The generic _tag read-filter in get_cfg_2stages is retained as defensive backward-compat for any stale user CSV. The FlyDSL-vs-CK dispatch gates (is_flydsl1 and is_flydsl_available()) are untouched — those select the wrapper, not fallback config.

Config data

  • Removed the obsolete flydsl_fallback rows (372 rows) from the 7 tuned configs that carried them: dsv3_fp4, dsv4_fp8fp4, gptoss_fp8fp4, kimik2_fp4, kimik2_fp8fp4, minimax_m25_fp4, qwen3_5_397b_fp4. Only the tagged rows were deleted; all other rows and the _tag column are left byte-identical to minimize churn.

Test

  • op_tests/tuning_tests/test_online_tune.py::test_tag_filter de-hardcoded its sample tag value (still validates the retained _tag read-filter).

Behavior change

With FlyDSL mandatory, a tuned config that names a flydsl_ kernel now requires FlyDSL to be installed — there is no CK fallback substitution. If FlyDSL is absent, the FlyDSL dispatch gate is skipped as before, so a flydsl_-named config will no longer silently degrade to a CK kernel.

Testing

  • python -m py_compile on both edited modules — OK.
  • pytest op_tests/tuning_tests/test_online_tune.py -k tag_filter — passing.
  • Repo-wide grep for flydsl_fallback — no matches remain.

🤖 Generated with Claude Code

@coderfeli
coderfeli requested a review from a team July 1, 2026 01:06
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4025 --add-label <label>

@coderfeli
coderfeli requested a review from junhaha666 July 1, 2026 01:11
lalala-sh
lalala-sh previously approved these changes Jul 1, 2026

@lalala-sh lalala-sh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@coderfeli
coderfeli force-pushed the clean_moe_flydsl_fallback branch from 8060b28 to 3e9bd34 Compare July 7, 2026 11:47
@coderfeli
coderfeli merged commit eaef170 into main Jul 8, 2026
42 checks passed
@coderfeli
coderfeli deleted the clean_moe_flydsl_fallback branch July 8, 2026 03:49
Rohan138 added a commit to Rohan138/aiter that referenced this pull request Jul 9, 2026
…-V3/R1 (gfx950)

DeepSeek-R1-0528-MXFP4 calls the fused MoE with 256 routed experts / topk=8
(the shared expert is handled separately by the serving framework), but
dsv3_fp4_tuned_fmoe.csv keys the tuned FlyDSL entries at expert=257/topk=9
(256 routed + 1 shared). On gfx950 the runtime lookup misses and falls back
to the slow FlyDSL heuristic -- ~8% E2EL regression at low concurrency
(ROCM-25536; ROCm#3909).

Add expert=256/topk=8 entries for the model_dim=7168 routed-expert shapes
(inter_dim 256 [TP8] and 512 [TP4], all token buckets) by re-keying the
existing tuned rows. Expert count does not change the per-expert GEMM shape,
so the tuned kernel selection is unchanged; the rows now reflect the k_wave
fp4 gemm1 kernels from ROCm#3825 and the fallback-hint cleanup from ROCm#4025.

Perf/accuracy validated earlier on MI350X/gfx950, vLLM 0.21.0, DSR1-MXFP4 TP8
(pre-ROCm#3825 kernels): ck_moe_2stages token=1 43->30us; serve E2EL conc1
10846->9987ms; gsm8k 5-shot 0.965. Re-validation on the ROCm#3825-retuned kernels
is advisable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
valarLip pushed a commit that referenced this pull request Jul 11, 2026
…-V3/R1 (gfx950) (#3911)

DeepSeek-R1-0528-MXFP4 calls the fused MoE with 256 routed experts / topk=8
(the shared expert is handled separately by the serving framework), but
dsv3_fp4_tuned_fmoe.csv keys the tuned FlyDSL entries at expert=257/topk=9
(256 routed + 1 shared). On gfx950 the runtime lookup misses and falls back
to the slow FlyDSL heuristic -- ~8% E2EL regression at low concurrency
(ROCM-25536; #3909).

Add expert=256/topk=8 entries for the model_dim=7168 routed-expert shapes
(inter_dim 256 [TP8] and 512 [TP4], all token buckets) by re-keying the
existing tuned rows. Expert count does not change the per-expert GEMM shape,
so the tuned kernel selection is unchanged; the rows now reflect the k_wave
fp4 gemm1 kernels from #3825 and the fallback-hint cleanup from #4025.

Perf/accuracy validated earlier on MI350X/gfx950, vLLM 0.21.0, DSR1-MXFP4 TP8
(pre-#3825 kernels): ck_moe_2stages token=1 43->30us; serve E2EL conc1
10846->9987ms; gsm8k 5-shot 0.965. Re-validation on the #3825-retuned kernels
is advisable.

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants