Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(sm120): 接入 flashinfer b12x,为 sm_120/121 添加 NVFP4 MoE 算子支持
背景
在RTX Pro 5000(SM 12.0)上以 NVFP4精度运行 MoE 模型(如 Qwen3.5-35B-A3B-NVFP4)时,rtp-llm 现有的两条 NVFP4 MoE 路径在 sm_120/121 上暂时均不可用:
TrtllmFp4Executor(trtllm-gen cubin):trtllm-gen 预编译 cubin 仅覆盖 SM100 系列架构(SM100/103/110),SM120 上没有对应 MoE cubin。CutedslFp4Executor(CuTe DSL blockscaled,tcgen05):kernel 同样依赖 SM100 独有的 tcgen05 指令。flashinfer 已为 sm_120/121 专门提供b12x CuTe DSL fused MoE,但是其目前限制CUDA版本须>=13。经本地实验验证,该限制过于严格,实际上CUDA 12.9 已经可以对sm_120f架构进行正确编译。其执行、数值全部正确。本 PR 据此把 b12x 接入 rtp-llm。
改动内容
B12xFp4Executor(.../executors/b12x_fp4_executor.py),在其中添加ptach放宽cuda版本要求strategy/fp4.py、strategy/__init__.py、fused_moe/__init__.py、defs/type.py)device_impl.py)server_args/moe_group_args.py)测试
新增
.../executors/test/b12x_fp4_executor_test.py,对算子进行数值正确性单测:相关内容