Roadmap (2025 Q4)

[sricketts]

We would like feedback from the community on this rough plan for Q4. This is of course a work in progress, and we welcome feedback at any time. Please add comments below or on any specific issues. We will edit this description as plans change.

Also, if you're interested in contributing, feel free to dive into any of the unassigned issues!

Soul

These are the broad areas of focus for the quarter. Items in the roadmap below are tagged by “soul item”.

• [Testing]: Better testing and CI infrastructure to prevent build breaks and accuracy issues at the framework level
• [Model Optimization]: DeepSeek-R1, GPT-OSS, Qwen3, Qwen3-Next, MiniCPM4.1-8B, and others
• [API Usability]: API cleanup and refactoring for better user experience

October

• [Model Optimization] Update the routing for TRTLLMGEN to support kimi k2 and qwen #1831
• [Model Optimization] GPT-OSS perf improvements for max throughput case
• [Testing] Expanded CI coverage per-PR (ability to trigger tests on NVIDIA-internal test infrastructure, including various Blackwell devices)
• [Model Optimization] Non-gated MoE with squared ReLU activation Feature: Support Relu2 activation in fused MoE #1954
• [Model Optimization] [Perf] FP4 MoE on B200 (latency) #1734, perf: Speed up fp4 quantization for small batch with swizzling for cutlass MoE #2025
• [Model Optimization] Add RoPE, RoPE+Q, RoPE+Q+KVCacheUpdate fused kernels for MLA/GQA/MHA
  • MLA RoPE + quantization fused kernel: shape generalization for MHA / GQA #1924
  • feat: Add flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache (fused RoPE + Q + KV cache, supports MLA/GQA/MHA)  #2037
• [API Usability] Support checks PoC #1809
• [API Usability] refactor: using tvm-ffi for multi-platform bindings #1641

November

• [Feature Request] Support all head dims in BatchQKApplyRotaryPosIdsCosSinCache #2104
• [feature request] add optimal gb200 moe comm kernels #2094
• [Feature Request] "auto" backend for mm_fp4 #1722
• FlashInfer 0.2.3+ does not support per-request generators #1104
• [Model Optimization] DSR1 improvements (details TBD)
• [Model Optimization] [Feature Request] Gated Delta Net #1690
• [Testing] Initial integration testing: e2e functional sanity checks
• [Testing] [API Usability] Minimal example of deploying LLM with flashinfer APIs (e.g. through gpt-fast). #1811
• [Model Optimization] Cosmos Reasoning 7B (details TBD)
• [Model Optimization] MXFP4 gemm perf improvements
• [API Usability] Support FP8-qkv FP8/FP4-output trtllm-gen in FlashInfer prefill/decode wrapper
• [API Usability] Unify qk_scale and o_scale Behavior Between trtllm-gen Attention and flashinfer-jit Attention
• [API Usability] Fused MoE general improvements, including (but not limited to):
  • RFC: Flashinfer MoE Wrappers for vLLM Integration
  • DeepSeek FP8 Block Scaling is not yet implemented in CUTLASS fused_moe #1669
• [API Usability] Inaccurate API Docstrings for Attention Prefill #1709
• [API Usability] Support checks: follow up to scale Support checks PoC #1809 across the library
• [Testing] Add comprehensive test xfails tracking system and analysis report #1733
• [Model Optimization] Native Sparse Attention
• [Model Optimization] [Feature Request] TopK Sparse Attention #1691
• [API Usability] [Feature Request] "auto" backend for mm_fp4 #1722
• [Model Optimization] [Perf] FP4 GEMM on B200 #1732
• API Usability] Unifying quantization related modules (fp4 quantize/quantize) silu_and_mul nvfp4 quanization fusion rework #1927

December

• [API Usability] Attention API consolidation
• [Testing] Improved unit testing based on escape analysis
• [Testing] Improved integration testing based on escape analysis

[sricketts]

We're working on prioritizing ideas from [Performance]: Custom fused kernel tracking · Issue #25179 · vllm-project/vllm and will add to the roadmap once we have a more concrete plan.

[rainj-me]

Is there any plan for Cute DSL support on following kernels ?

1. RoPE + Set KvCache FP8/FP4
2. Fuse Attention on FP8/FP4
3. General MoE on FP8/FP4
4. Fuse MoE for DS R1, Qwen3 models on FP8/FP4

[sricketts]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

[rainj-me]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

@sricketts , thanks for following up. I'm trying to propose these kernels due to

1. flashinfer already has gemm kernel base on Cute DSL, so it is not introducing a brand new implementations, just to extend the existing functions
2. some existing kernels/solutions only works for specific model and quantization, it is hard to expand to models or quantization

[sricketts]

Is there any plan for Cute DSL support on following kernels ?

@rainj-me when choosing how to implement kernels in FlashInfer, we consider a number of factors (e.g. perf, maintainability, open vs closed source, etc). Can you some context for your question -- why do you want CuTe DSL implementations of these op patterns (as opposed to some other implementation choice)?

@sricketts , thanks for following up. I'm trying to propose these kernels due to

1. flashinfer already has gemm kernel base on Cute DSL, so it is not introducing a brand new implementations, just to extend the existing functions
2. some existing kernels/solutions only works for specific model and quantization, it is hard to expand to models or quantization

Thanks -- let me rephrase to make sure I understand:

• as a general comment you would like to see more open source implementations, so you can extend them for your use cases (e.g. model variations or different quantization recipes).
• since there is already a CuTeDSL gemm kernel in flashinfer, you think that would be a natural choice for the target language for FlashInfer OSS kernels
• I'm guessing you also just like CuTeDSL and think it's a good choice for extensibility :)

Did I get that right?

[samuellees]

@sricketts Could flashinfer provide wheel with AOT kernels? Thanks

[yzh119]

Hi @samuellees instead of calling aot wheel, we call it flashinfer-jit-cache package now, implemented in #1726 (pending some final testing to merge). We also provide another pypi package https://pypi.org/project/flashinfer-cubin/ for cubin kernels.

If only flashinfer-python is installed, everything will be jit compiled.
If user want to have full coverage of pre-built kernels:

pip install -U flashinfer-python flashinfer-cubin
pip install -U flashinfer-jit-cache -i https://flashinfer.ai/whl/cu130 # for cuda 130

[yzh119]

@rainj-me for open-source kernels, cute-dsl will be the default choice for future kernel development (they are still not the default choice for all kernels yet because some bugs/compatibility with earlier architectures).

[yonikremer]

"[Testing]: Better testing and CI infrastructure to prevent build breaks and accuracy issues at the framework level"
I must add that today the tests are very slow, for example the sampling tests contain 900 tests, a lot of them on very large tensors, which makes testing your changes quite hard.
I think a better testing speed is neede for dev productivity and experiance.

[ZelinMa557]

Will flashinfer support fp8 attention on ada?