[AMD] [GLM-5.3-Flash Day 0] Build the fused DSA k-pool top-k JIT kernel on HIP - #38542
Closed
Raiden-Makoto wants to merge 6 commits into
Closed
Raiden-Makoto wants to merge 6 commits into
Raiden-Makoto wants to merge 6 commits into
Conversation
Raiden-Makoto
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
September 8, 2026 17:25
Raiden-Makoto
marked this pull request as draft
September 8, 2026 18:03
Raiden-Makoto
marked this pull request as ready for review
September 8, 2026 19:28
Raiden-Makoto
force-pushed
the
RM/reopen-37563
branch
from
September 8, 2026 19:36
8d704fa to
cfa366e
Compare
jit/csrc/dsa/kpool_topk_transform.cuh includes <cuda_fp16.h> directly and reaches ::cudaFuncSetAttribute through no alias, so hipcc cannot compile it. On gfx950 the JIT build fails with "fatal error: 'cuda_fp16.h' file not found", and after that with "no member named 'cudaFuncSetAttribute' in the global namespace". This is a hard stop rather than a slow path. GLM-5.3-Flash pools index_topk=2048 over index_kpool=4, so group_topk is 512, and kpool_fp8_index routes every pooled budget in (128, 160, 192, 224, 256, 512) to this JIT kernel -- the fast_topk_v2 fallback covers group_topk 2048 alone. The indexer cannot run at all on ROCm. Give the fp16 include a USE_ROCM arm, matching jit/csrc/elementwise/fused_metadata_copy.cuh in the same tree, and add cudaFuncSetAttribute to the USE_ROCM alias list in sgl_kernel/utils.cuh beside the existing cudaGetLastError and cudaDeviceGetAttribute entries. Both CUDA lines stay untouched. Add a focused parity test. The kernel has no direct coverage on any platform; CUDA reaches it end to end through test/registered/models_e2e/test_glm53_flash_b200.py, which serves GLM-5.3-Flash and therefore JIT-compiles it every commit, while AMD has no equivalent. Registered on the mi35x per-PR suite only.
Raiden-Makoto
force-pushed
the
RM/reopen-37563
branch
from
September 8, 2026 19:41
cfa366e to
c1bb21c
Compare
Contributor
Author
|
Closing this replacement because @Jacob0226 will reopen and maintain the original PR, #37563. Please continue review and CI there. |
Contributor
|
Thanks @Raiden-Makoto. Reopening as #39339 rather than #37563: #37563's base branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replacement for #37563, which GitHub closed when its deleted support base was merged into main. The original author and review history remain credited there.
jit/csrc/dsa/kpool_topk_transform.cuhincludes<cuda_fp16.h>directly and reaches::cudaFuncSetAttributethrough no alias, so hipcc cannot compile it. This is a hard stop on ROCm rather than a slow path: GLM-5.3-Flash poolsindex_topk=2048overindex_kpool=4, givinggroup_topk=512, andkpool_fp8_indexroutes every pooled budget in(128, 160, 192, 224, 256, 512)to this JIT kernel — thefast_topk_v2fallback coversgroup_topk=2048alone, so the indexer cannot run at all.__half,__float2half_rn,__half_as_ushort<cuda_fp16.h>include aUSE_ROCMbranch that reaches for<hip/hip_fp16.h>, matchingjit/csrc/elementwise/fused_metadata_copy.cuhin the same tree. The CUDA line is untouchedcudaFuncSetAttributesgl_kernel/utils.cuh, beside the shim's existingcudaGetLastErrorandcudaDeviceGetAttributeentries, rather than guarding one call siteScope
One include wrapped in
USE_ROCMbranches, two aliases added to the shim's existingUSE_ROCMlist. Every CUDA line is unchanged context in the diff and resolves exactly as before. No kernel body, launch configuration, tile size or numeric path moves; no environment variable, no architecture gate.fast_kpool_topk_transform_fusedhas no direct test on any platform. CUDA reaches it end to end through test_glm53_flash_b200.py, which serves GLM-5.3-Flash and therefore JIT-compiles it every commit; AMD has no equivalent, so this adds focused unit coverage on the mi35x per-PR suite.Test plan
Combined main-based integration (
RM/glm53-day0-main-integrationtested at9d0d62b744; current integration head4e2f84ff3c; TP8/TP4 validation predates the above-cap guard, whose below-cap path is unchanged, pinned AITERd9e5ef7ce0, fresh caches): TP8 GSM8K 97.50% in 87.0s with 0.00% errors; TP4 GSM8K 96.97% in 320.2s with 0.00% errors. Decode CUDA graphs were enabled in both runs.Recovery base:
main@30e7a3072d.Recovery validation on MI355X: scoped pre-commit passed; 4 fresh-cache HIP JIT tests passed.
rocm/sgl-dev:v0.5.18-rocm720-mi35x-20260901, the image familypr-test-amd-rocm720.ymlresolves to, onxinyuan/glm-5.3-flash-support@515e865189with no other PRs applied. Both builds run the same new test file; the shim aliases are the only variable. JIT caches were cleared between builds, and the build hashes differ, so neither result is a stale artifact.group_topk128 and 512 alike, on'cuda_fp16.h' file not foundandno member named 'cudaFuncSetAttribute' in the global namespaceCI States
Latest PR Test (Base): ⏳ Run #34768730151
Latest PR Test (Extra): ❌ Run #34768729972
Latest PR Test (AMD ROCm 10): ❌ Run #34768729836