Conversation
jit/csrc/dsa/kpool_topk_transform.cuh includes <cuda_fp16.h> directly and reaches ::cudaFuncSetAttribute through no alias, so hipcc cannot compile it. On gfx950 the JIT build fails with "fatal error: 'cuda_fp16.h' file not found", and after that with "no member named 'cudaFuncSetAttribute' in the global namespace". This is a hard stop rather than a slow path. GLM-5.3-Flash pools index_topk=2048 over index_kpool=4, so group_topk is 512, and kpool_fp8_index routes every pooled budget in (128, 160, 192, 224, 256, 512) to this JIT kernel -- the fast_topk_v2 fallback covers group_topk 2048 alone. The indexer cannot run at all on ROCm. Give the fp16 include a USE_ROCM arm, matching jit/csrc/elementwise/fused_metadata_copy.cuh in the same tree, and add cudaFuncSetAttribute to the USE_ROCM alias list in sgl_kernel/utils.cuh beside the existing cudaGetLastError and cudaDeviceGetAttribute entries. Both CUDA lines stay untouched. Add a focused parity test. The kernel has no direct coverage on any platform; CUDA reaches it end to end through test/registered/models_e2e/test_glm53_flash_b200.py, which serves GLM-5.3-Flash and therefore JIT-compiles it every commit, while AMD has no equivalent. Registered on the mi35x per-PR suite only. Co-authored-by: Cursor <cursoragent@cursor.com>
Jacob0226
force-pushed
the
jacob/glm53-rocm-kpool-hip
branch
from
September 2, 2026 08:18
f94e9fa to
261d655
Compare
Jacob0226
marked this pull request as ready for review
September 2, 2026 10:29
Jacob0226
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
September 2, 2026 10:29
This was referenced Sep 8, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jit/csrc/dsa/kpool_topk_transform.cuhincludes<cuda_fp16.h>directly and reaches::cudaFuncSetAttributethrough no alias, so hipcc cannot compile it. This is a hard stop on ROCm rather than a slow path: GLM-5.3-Flash poolsindex_topk=2048overindex_kpool=4, givinggroup_topk=512, andkpool_fp8_indexroutes every pooled budget in(128, 160, 192, 224, 256, 512)to this JIT kernel — thefast_topk_v2fallback coversgroup_topk=2048alone, so the indexer cannot run at all.__half,__float2half_rn,__half_as_ushort<cuda_fp16.h>include aUSE_ROCMbranch that reaches for<hip/hip_fp16.h>, matchingjit/csrc/elementwise/fused_metadata_copy.cuhin the same tree. The CUDA line is untouchedcudaFuncSetAttributesgl_kernel/utils.cuh, beside the shim's existingcudaGetLastErrorandcudaDeviceGetAttributeentries, rather than guarding one call siteScope
One include wrapped in
USE_ROCMbranches, two aliases added to the shim's existingUSE_ROCMlist. Every CUDA line is unchanged context in the diff and resolves exactly as before. No kernel body, launch configuration, tile size or numeric path moves; no environment variable, no architecture gate.fast_kpool_topk_transform_fusedhas no direct test on any platform. CUDA reaches it end to end through test_glm53_flash_b200.py, which serves GLM-5.3-Flash and therefore JIT-compiles it every commit; AMD has no equivalent, so this adds focused unit coverage on the mi35x per-PR suite.Test plan
rocm/sgl-dev:v0.5.18-rocm720-mi35x-20260901, the image familypr-test-amd-rocm720.ymlresolves to, onxinyuan/glm-5.3-flash-support@515e865189with no other PRs applied. Both builds run the same new test file; the shim aliases are the only variable. JIT caches were cleared between builds, and the build hashes differ, so neither result is a stale artifact.group_topk128 and 512 alike, on'cuda_fp16.h' file not foundandno member named 'cudaFuncSetAttribute' in the global namespaceCI States
Latest PR Test (Base): ❌ Run #33619800731
Latest PR Test (Extra): ❌ Run #33628108935
Latest PR Test (AMD ROCm 7.2): ❌ Run #33619800720