[fp8] Select SGLang FP8 block quant kernel to match inference - #1182
Conversation
bcd5394 to
4aee1fc
Compare
4aee1fc to
b6ccb11
Compare
yueming-yuan
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates the per_block_cast_to_fp8 function from the sglang library into the FP8 quantization workflow. It introduces a new internal helper, _blockwise_cast_to_fp8, which conditionally utilizes the sglang implementation when the weight block size is (128, 128) and falls back to the existing Triton-based kernel otherwise. Additionally, the sglang utility is safely imported with a fallback to None to ensure compatibility. I have no feedback to provide as there were no review comments to evaluate.
maocheng23
left a comment
maocheng23
left a comment
There was a problem hiding this comment.
Approved to unblock, but better address the message
| if _get_scale_format(args, name, weight_block_size) == "ue8m0": | ||
| qweight, scale = quant_weight_ue8m0(weight, weight_block_size=weight_block_size) | ||
| scale = transform_scale_ue8m0(scale, mn=qweight.shape[-2]) | ||
| elif per_block_cast_to_fp8 is not None and list(weight_block_size) == [128, 128]: |
There was a problem hiding this comment.
This looks very hacky, any better ways to do this?
There was a problem hiding this comment.
added a TODO comment. will fix these issues in later low-precision refactor
Added a TODO comment to address the hacky implementation for [128, 128] weight block size.
2 participants
No description provided.