[None][feat] Update the logic of FMHA JIT path - #14291
Conversation
|
/bot run --disable-fail-fast |
|
PR_Github #49109 [ run ] triggered by Bot. Commit: |
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
dd43341 to
dd46263
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #49189 [ run ] triggered by Bot. Commit: |
|
PR_Github #49109 [ run ] completed with state |
|
PR_Github #49189 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49301 [ run ] triggered by Bot. Commit: |
|
PR_Github #49301 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #49338 [ run ] triggered by Bot. Commit: |
|
PR_Github #49338 [ run ] completed with state
|
|
/bot run |
|
PR_Github #49384 [ run ] triggered by Bot. Commit: |
|
PR_Github #49384 [ run ] completed with state |
|
✅ LFS objects already in storage (2982 files) — no sync needed. These LFS-tracked files are already present in this repository's LFS storage:
|
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
Rebases the TokenSpeed K2.6 evaluation onto upstream/main (PR NVIDIA#14291, FMHA JIT namespace fix) and reports the resulting clean TS vs TRTLLM A/B numbers. Pre-rebase, the TRTLLM baseline crashed during warmup with NVRTC compilation failures on the fmhaSm103aKernel ...HQk576HV512 ...ForGen family; the patch in PR NVIDIA#14291 (cutlass:: -> trtllm::dev::) restores it. Results (B300, BF16-KV patched K2.6): - TP4 1k/1k conc=1: TRTLLM 158.5 / TS 152.8 tok/s (-3.6%) - TP8 1k/1k conc=1: TRTLLM 182.2 / TS 169.3 tok/s (-7.1%) - TP4 8k/1k conc=1: TRTLLM 152.0 / TS 146.9 tok/s (-3.4%) - TP4 1k/1k conc=16: TRTLLM 1239.3 / TS 1246.8 tok/s (+0.6%, tied) Files: - new: phase4-rebased-ab.md, nvrtc-rebase-verify.md (verification + writeup), bench-config_base.yml (TRTLLM sidecar), scripts/run_bench_v3.sh (A/B driver targeting the rebased build) - updated: phase4-summary.md (rebased context + NVBugs link), nvbug-draft-nvrtc-baseline.md (archived banner, fix landed upstream) - removed: unused bench-60k1k_*.yml, bench-1k1k_tp4_conc16_attndp.yml, bench-1k1k_tp4_conc16_mtp3.yml, bench-8k1k_tp4_conc1_mtp3.yml and early prototype code/{tokenspeed_mla,test_tokenspeed_mla}.py (superseded by the in-tree backend at tensorrt_llm/_torch/attention_backend/tokenspeed_mla*.py) Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>
3 participants
@coderabbitai summary
Description
In this PR:
Update all the cubins/headers/libs from trtllm-gen ToT main with full dynamic sparse kernels and zero sparse MQA/GQA cubins:
Update the logic of
shouldUseNvrtcTest Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.