Deepseek r1 fp8matmul

[xuechendi]

Decode latency improved ~1.5x

Default we disabled this feature

When enabling with original static fp8 path, this feature will remove the dequant/quant between kv_cache and matmul_qk.

VLLM_USE_FP8_MATMUL=true \

When working with INC static fp8, this feature can further enable fp8 pipeline PA since we leverage INC to provide batch2block_matmul scaling. Also, MOE will be faster due to scalar MOE from INC.

QUANT_CONFIG=scripts/inc_quant_with_fp8kv_config.json \
VLLM_REQUANT_FP8_INC=1 \
VLLM_ENABLE_RUNTIME_DEQUANT=1 \
VLLM_USE_FP8_MATMUL=true \

** with bs224_blocks7296 **
fp8 matmul is

[kwisniewski98]

Are those env vars necessary? Why VLLM_PT_PROFILE cannot be used for profiling?

[xuechendi]

removed profiling in this PR. The 'VLLM_PT_PROFILE' produce dummy input which can't fully test MOE which makes profiling faster.