Repository navigation
[FlyDSL] Support runtime softmax scale in gfx950 FP8 FMHA - #5405
Merged
Merged
Conversation
gbyu-amd
force-pushed
the
gbyu/flydsl-fmha-softmax-scale
branch
from
September 10, 2026 14:50
ade9a4a to
3407084
Compare
binding7012
reviewed
Sep 11, 2026
Contributor
|
LGTM |
coderfeli
approved these changes
Sep 14, 2026
This was referenced Sep 14, 2026
sammysun0711
pushed a commit
to sammysun0711/aiter
that referenced
this pull request
Sep 16, 2026
* [FlyDSL] Support runtime softmax scale in gfx950 FP8 FMHA * [FlyDSL] Share FP8 FMHA scale validation and remove redundant launch argument Signed-off-by: Xiake Sun <xiake.sun@amd.com>
RElbers
pushed a commit
that referenced
this pull request
Sep 25, 2026
* [FlyDSL] Support runtime softmax scale in gfx950 FP8 FMHA * [FlyDSL] Share FP8 FMHA scale validation and remove redundant launch argument
RElbers
pushed a commit
that referenced
this pull request
Sep 25, 2026
* [FlyDSL] Support runtime softmax scale in gfx950 FP8 FMHA * [FlyDSL] Share FP8 FMHA scale validation and remove redundant launch argument
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gfx950 FlyDSL FP8 FMHA kernel currently fixes the attention scale to
1 / sqrt(D). Consumers using a different layer scale, such as YaRN, must fold the correction into a quantization descale, and the AITER dispatch rejects nondefault scales.Add
softmax_scale=Noneto the direct FP8 API, using the BF16 API's parameter name and default. Pass it as a runtime FP32 scalar and applysoftmax_scale * q_descale * k_descaleto the logits inside FMHA. Dense and varlen dispatch and the large-tensor batch-slicing path forward the scale. This needs no descale mutation or additional GPU kernel launch, and scale values do not enter the build-cache key. Output and LSE use the same scale.The existing online softmax requires a positive, finite scale: it takes maxima and applies masking before scaling. The direct API rejects zero, negative and non-finite scales; dispatch returns
Nonefor those configurations. Supporting them would require a separate change to the softmax algorithm.This PR targets
mainand contains only FMHA scale support, dispatch changes and tests. The review follow-up removes the redundanthead_dim_runtimelaunch argument and shares softmax-scale validation between the dispatch gate and direct FP8 API.Validation:
a859b8271): 24 CPU tests passed, 252 GPU tests skipped, with HIP/ROCR/CUDA device visibility disabled. Fake-tensor tests exercise the direct API and dense/varlen dispatch for default/custom/invalid scales; launch-boundary tests check default and custom scales in both run and compile argument preparation for D=128/192. No GPU execution or GPU compilation was performed. Black, Ruff andgit diff --checkpass.git diff --checkpass; all 252 FMHA tests collect successfully with GPUs hidden (AITER_AOT_IMPORT=1). Collection is not a correctness run.git range-diffconfirmed the feature patch was unchanged; all five changed files were byte-identical to the previously GPU-validated PR head. GPU tests were not repeated during the active TP8 workload.softmax_scaleforwarding: 112 tests passed.